Hi, we'll try to answer every question.
Regarding this:
Unable to install python packages from tar.gz files. The python package refers to other dependencies and unable to build wheels for installation as well. (Network restrictions)
Here you have some general solutions:
- Generate image locally with everything needed, upload and run on the platform with uDocker.
- Generate the wheel locally, upload and install on the platform. To generate the package locally you can use: docker run --rm -it -v /tmp:/tmp/host-tmp python:3.10 bash. This way you will have the same python version as on the platform and the generated package will be compatible.
Regarding this:
Unable to copy file from persistent-home to /home of the ubuntu-python-pytorch:latest-cuda to do custom installations.
There's no restriction there, can you send a screenshot or the specific error please?
About this:
Can't use udocker for training process using prepared docker image as well (no option/examples showing how to utilize GPU with udocker within a jobman job) .
Here you have the uDocker example https://github.com/chaimeleon-eu/workstation-images/blob/main/usage-guide.md#running-an-image-with-udocker
jobman submit -- "udocker load -i ~/persistent-home/alpine-3.9.tar.gz \
&& udocker run --rm -v /home/chaimeleon/persistent-home -v /home/chaimeleon/datasets -v /mnt/datalake \
alpine:3.9 ls -lh /home/chaimeleon/persistent-home"
To have GPU just add the option "-r " as shown in the GPU example: https://github.com/chaimeleon-eu/workstation-images/blob/main/usage-guide.md#jobman-client-tool
jobman submit -i ubuntu-python:latest-cuda -r small-gpu -- nvidia-smi
And joining the two examples:
jobman submit -r small-gpu -- "udocker load -i ~/persistent-home/alpine-3.9.tar.gz \
&& udocker run --rm -v /home/chaimeleon/persistent-home -v /home/chaimeleon/datasets -v /mnt/datalake \
alpine:3.9 ls -lh /home/chaimeleon/persistent-home"
No Internet connection is a restriction to install software, we know that, but it is a condition of the project that data cannot leave the platform (if we allow Internet connection, data could be uploaded to the Internet).
Tools like pip make it very easy to install software when there is a direct Internet connection and much more difficult when there is not, but it is just a matter of uploading everything you need from your computer. There are several packages already uploaded to ~/persistent-shared-folder/pypi.org/, so you can install them that way:
pip install --no-index --find-links ~/persistent-shared-folder/pypi.org pyradiomics
pip install --no-index --find-links ~/persistent-shared-folder/pypi.org lifelines
pip install --no-index --find-links ~/persistent-shared-folder/pypi.org catboost
pip install --no-index --find-links ~/persistent-shared-folder/pypi.org pytorchvideo
Hope we could help!
