Failure on submission and no logs

Hi there,

I'm getting the following error when submitting in phase 2: - The algorithm failed on one or more cases.

This same algorithm worked fine when submitting to phase 1, and now it failed after approx 1h runnning (so it isn't a timeout). I have no more information than this to understand what is the problem with it. Can you send me the log, so that I can fix the issue?

Regards, Nuno

Hi Nuno,

This is the stderr of your last submission: 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:35: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 eps=np.finfo(np.float).eps, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:597: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 eps=np.finfo(np.float).eps, copy_X=True, fit_path=True, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:836: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 eps=np.finfo(np.float).eps, copy_X=True, fit_path=True, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:862: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 eps=np.finfo(np.float).eps, positive=False): 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1097: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1344: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1480: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 eps=np.finfo(np.float).eps, copy_X=True, positive=False): 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:152: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 precompute=False, eps=np.finfo(np.float).eps, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:320: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 eps=np.finfo(np.float).eps, random_state=None, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/linear_model/randomized_l1.py:580: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 eps=4 * np.finfo(np.float).eps, n_jobs=None, 2022-02-21T20:35:24+00:00 /home/algorithm/.local/lib/python3.7/site-packages/sklearn/decomposition/online_lda.py:31: DeprecationWarning: 'np.float' is a deprecated alias for the builtin 'float'. To silence this warning, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64' here. 2022-02-21T20:35:24+00:00 Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations 2022-02-21T20:35:24+00:00 EPS = np.finfo(np.float).eps 2022-02-21T20:35:35+00:00 0%| | 0/234 [00:00<?, ?it/s] 1%|▏ | 3/234 [00:00<00:08, 25.74it/s] 3%|▎ | 6/234 [00:00<00:08, 27.32it/s] 4%|▍ | 9/234 [00:00<00:10, 21.97it/s] 5%|▌ | 12/234 [00:00<00:09, 22.98it/s] 6%|▋ | 15/234 [00:00<00:09, 24.32it/s] 8%|▊ | 18/234 [00:00<00:10, 21.44it/s] 9%|▉ | 22/234 [00:00<00:08, 25.04it/s] 11%|█ | 25/234 [00:01<00:09, 22.63it/s] 12%|█▏ | 28/234 [00:01<00:08, 23.60it/s] 13%|█▎ | 31/234 [00:01<00:08, 24.35it/s] 15%|█▍ | 34/234 [00:01<00:07, 25.22it/s] 16%|█▌ | 37/234 [00:01<00:08, 22.09it/s] 17%|█▋ | 40/234 [00:01<00:08, 23.40it/s] 18%|█▊ | 43/234 [00:01<00:09, 20.88it/s] 20%|██ | 47/234 [00:02<00:07, 23.45it/s] 21%|██▏ | 50/234 [00:02<00:08, 21.87it/s] 23%|██▎ | 54/234 [00:02<00:07, 24.58it/s] 24%|██▍ | 57/234 [00:02<00:08, 22.04it/s] 26%|██▌ | 60/234 [00:02<00:07, 23.03it/s] 27%|██▋ | 63/234 [00:02<00:07, 24.27it/s] 29%|██▊ | 67/234 [00:02<00:06, 25.93it/s] 30%|██▉ | 70/234 [00:03<00:07, 22.83it/s] 32%|███▏ | 74/234 [00:03<00:06, 23.28it/s] 33%|███▎ | 77/234 [00:03<00:06, 24.73it/s] 35%|███▍ | 81/234 [00:03<00:05, 27.34it/s] 36%|███▌ | 84/234 [00:03<00:05, 26.93it/s] 38%|███▊ | 88/234 [00:03<00:04, 29.49it/s] 39%|███▉ | 92/234 [00:03<00:04, 30.81it/s] 41%|████ | 96/234 [00:03<00:04, 27.85it/s] 42%|████▏ | 99/234 [00:04<00:04, 27.76it/s] 44%|████▎ | 102/234 [00:04<00:04, 28.01it/s] 45%|████▌ | 106/234 [00:04<00:04, 30.75it/s] 47%|████▋ | 110/234 [00:04<00:04, 27.20it/s] 49%|████▊ | 114/234 [00:04<00:04, 29.46it/s] 50%|█████ | 118/234 [00:04<00:03, 31.02it/s] 52%|█████▏ | 122/234 [00:04<00:03, 28.60it/s] 53%|█████▎ | 125/234 [00:04<00:04, 23.48it/s] 55%|█████▍ | 128/234 [00:05<00:04, 23.98it/s] 56%|█████▋ | 132/234 [00:05<00:03, 26.69it/s] 58%|█████▊ | 135/234 [00:05<00:04, 24.17it/s] 59%|█████▉ | 139/234 [00:05<00:03, 26.37it/s] 61%|██████ | 142/234 [00:05<00:03, 27.07it/s] 62%|██████▏ | 145/234 [00:05<00:03, 26.36it/s] 63%|██████▎ | 148/234 [00:05<00:03, 23.25it/s] 65%|██████▍ | 152/234 [00:06<00:03, 26.54it/s] 66%|██████▌ | 155/234 [00:06<00:03, 26.30it/s] 68%|██████▊ | 159/234 [00:06<00:02, 26.24it/s] 70%|██████▉ | 163/234 [00:06<00:02, 28.67it/s] 71%|███████ | 166/234 [00:06<00:02, 28.34it/s] 72%|███████▏ | 169/234 [00:06<00:02, 27.54it/s] 74%|███████▎ | 172/234 [00:06<00:02, 24.27it/s] 75%|███████▍ | 175/234 [00:06<00:02, 24.73it/s] 76%|███████▌ | 178/234 [00:06<00:02, 25.75it/s] 78%|███████▊ | 182/234 [00:07<00:01, 28.85it/s] 79%|███████▉ | 185/234 [00:07<00:01, 27.75it/s] 81%|████████ | 189/234 [00:07<00:01, 30.69it/s] 82%|████████▏ | 193/234 [00:07<00:01, 29.48it/s] 84%|████████▍ | 197/234 [00:07<00:01, 28.71it/s] 86%|████████▌ | 201/234 [00:07<00:01, 28.12it/s] 87%|████████▋ | 204/234 [00:07<00:01, 27.81it/s] 88%|████████▊ | 207/234 [00:07<00:00, 27.61it/s] 90%|█████████ | 211/234 [00:08<00:00, 30.19it/s] 92%|█████████▏| 215/234 [00:08<00:00, 27.77it/s] 93%|█████████▎| 218/234 [00:08<00:00, 27.15it/s] 95%|█████████▍| 222/234 [00:08<00:00, 28.22it/s] 96%|█████████▌| 225/234 [00:08<00:00, 28.42it/s] 98%|█████████▊| 229/234 [00:08<00:00, 26.42it/s] 99%|█████████▉| 232/234 [00:08<00:00, 27.10it/s] 100%|██████████| 234/234 [00:08<00:00, 26.07it/s] 2022-02-21T20:35:35+00:00 Traceback (most recent call last): 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main 2022-02-21T20:35:35+00:00 "__main__", mod_spec) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code 2022-02-21T20:35:35+00:00 exec(code, run_globals) 2022-02-21T20:35:35+00:00 File "/opt/algorithm/process.py", line 475, in <module> 2022-02-21T20:35:35+00:00 airogs_algorithm().process() 2022-02-21T20:35:35+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/evalutils/evalutils.py", line 183, in process 2022-02-21T20:35:35+00:00 self.process_cases() 2022-02-21T20:35:35+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/evalutils/evalutils.py", line 191, in process_cases 2022-02-21T20:35:35+00:00 self._case_results.append(self.process_case(idx=idx, case=case)) 2022-02-21T20:35:35+00:00 File "/opt/algorithm/process.py", line 416, in process_case 2022-02-21T20:35:35+00:00 results = self.predict(images=torch.vstack(images)) 2022-02-21T20:35:35+00:00 File "/opt/algorithm/process.py", line 437, in predict 2022-02-21T20:35:35+00:00 batch_predictions = self.model.forward(images).detach().cpu().float().numpy() 2022-02-21T20:35:35+00:00 File "/opt/algorithm/process.py", line 292, in forward 2022-02-21T20:35:35+00:00 x = self.model(x) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl 2022-02-21T20:35:35+00:00 return forward_call(*input, **kwargs) 2022-02-21T20:35:35+00:00 File "/opt/algorithm/process.py", line 142, in forward 2022-02-21T20:35:35+00:00 x = self.forward_features(x) 2022-02-21T20:35:35+00:00 File "/opt/algorithm/process.py", line 138, in forward_features 2022-02-21T20:35:35+00:00 x = self.stages[i](x) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl 2022-02-21T20:35:35+00:00 return forward_call(*input, **kwargs) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 141, in forward 2022-02-21T20:35:35+00:00 input = module(input) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl 2022-02-21T20:35:35+00:00 return forward_call(*input, **kwargs) 2022-02-21T20:35:35+00:00 File "/opt/algorithm/process.py", line 68, in forward 2022-02-21T20:35:35+00:00 x = self.act(x) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl 2022-02-21T20:35:35+00:00 return forward_call(*input, **kwargs) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 652, in forward 2022-02-21T20:35:35+00:00 return F.gelu(input) 2022-02-21T20:35:35+00:00 File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1556, in gelu 2022-02-21T20:35:35+00:00 return torch._C._nn.gelu(input) 2022-02-21T20:35:35+00:00 RuntimeError: CUDA out of memory. Tried to allocate 4.11 GiB (GPU 0; 14.76 GiB total capacity; 11.03 GiB already allocated; 2.57 GiB free; 11.04 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Best regards, Coen

Thank you Coen for posting the logs. Let's see if I can fix this.

Regards, Nuno