pytorch version and GPU support

pytorch version and GPU support  

  By: tboissin on Sept. 1, 2024, 1:50 p.m.

Hello,

I was wondering if we could know some informations about the GPUs used for evaluation (model and VRAM). For the backstory, I'm having issues with my submission ( The algorithm failed on one or more cases. I would be glad to have more informations ). The "Try-out algorithm" also fail, and the details tells me that "No logs found on stderr" and "No logs found on stdout". The only thing I can see is that GPU usage is down to 0%. My container is based on pytorch-2.1.2-cuda11.8-cudnn8-runtime, and I suspect this configuration to be compatible with the instances used for evaluation.

Re: pytorch version and GPU support  

  By: aneeqzia_isi on Sept. 1, 2024, 9:24 p.m.

Hi,

The evaluation container has access to 8 CPU, 32 GB RAM and 1 T4 GPU.

Following is the error of your submission:

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

Below is the detailed logs leading to the error:

2024-09-01T09:17:36.161000+00:00 /opt/conda/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be removed in 0.17. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional. 2024-09-01T09:17:36.161000+00:00 warnings.warn( 2024-09-01T09:17:38.162000+00:00 Could not find an int in the string 'endoscopic-robotic-surgery-video'. 2024-09-01T09:17:38.162000+00:00 Traceback (most recent call last): 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-09-01T09:17:38.162000+00:00 return _run_code(code, main_globals, None, 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code 2024-09-01T09:17:38.162000+00:00 exec(code, run_globals) 2024-09-01T09:17:38.162000+00:00 File "/opt/algorithm/process.py", line 264, in 2024-09-01T09:17:38.162000+00:00 SurgVU_classify().process() 2024-09-01T09:17:38.162000+00:00 File "/home/algorithm/.local/lib/python3.10/site-packages/evalutils/evalutils.py", line 183, in process 2024-09-01T09:17:38.162000+00:00 self.process_cases() 2024-09-01T09:17:38.162000+00:00 File "/home/algorithm/.local/lib/python3.10/site-packages/evalutils/evalutils.py", line 191, in process_cases 2024-09-01T09:17:38.162000+00:00 self._case_results.append(self.process_case(idx=idx, case=case)) 2024-09-01T09:17:38.162000+00:00 File "/opt/algorithm/process.py", line 164, in process_case 2024-09-01T09:17:38.162000+00:00 scored_candidates = self.predict(case.path) # video file > load evalutils.py 2024-09-01T09:17:38.162000+00:00 File "/opt/algorithm/process.py", line 244, in predict 2024-09-01T09:17:38.162000+00:00 preds = self.model(batch.unsqueeze(0)) 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-09-01T09:17:38.162000+00:00 return self._call_impl(args, kwargs) 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-09-01T09:17:38.162000+00:00 return forward_call(args, kwargs) 2024-09-01T09:17:38.162000+00:00 File "/home/algorithm/.local/lib/python3.10/site-packages/pytorchvideo/models/net.py", line 43, in forward 2024-09-01T09:17:38.162000+00:00 x = self.blocksidx 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-09-01T09:17:38.162000+00:00 return self._call_impl(*args, kwargs) 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-09-01T09:17:38.162000+00:00 return forward_call(args, kwargs) 2024-09-01T09:17:38.162000+00:00 File "/home/algorithm/.local/lib/python3.10/site-packages/pytorchvideo/models/stem.py", line 253, in forward 2024-09-01T09:17:38.162000+00:00 x = self.conv(x) 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl 2024-09-01T09:17:38.162000+00:00 return self._call_impl(args, kwargs) 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl 2024-09-01T09:17:38.162000+00:00 return forward_call(*args, kwargs) 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 610, in forward 2024-09-01T09:17:38.162000+00:00 return self._conv_forward(input, self.weight, self.bias) 2024-09-01T09:17:38.162000+00:00 File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 605, in _conv_forward 2024-09-01T09:17:38.162000+00:00 return F.conv3d( 2024-09-01T09:17:38.162000+00:00 RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

Re: pytorch version and GPU support  

  By: tboissin on Sept. 2, 2024, 8:30 a.m.

thank you very much !