Even if NLPL strives to make available a comprehensive set of ready-to-run sofware modules, users will at times want to install their own add-on components.
Python Virtual Environments on Saga
NLPL software modules that provide add-on components to Python are implemented as stacked virtual environments. This design requires that some care is taken when creating user-level virtual environments; most importantly, the correct base Python version needs to be used when creating and manipulating the virtual environment. In late 2019, we are building a first collection of NLPL Python add-ons on top of Python 3.7.2:
module purge; module load Python/3.7.2-GCCcore-8.2.0 python3 -m virtualenv foo . foo/bin/activate python3 -m pip install telegram
It is recommend to use
python3 -m virtualenv and the like instead
of short-hand commands like
this gives better control over exactly which Python interpreter is used.
For the above, for example, the following must hold (in late 2019):
type -path python3 /cluster/software/Python/3.7.2-GCCcore-8.2.0/bin/python3
One way of detecting the right base Python module to use for the creation of virtual environments is to inspect which version is loaded as prerequisites to a specific NLPL module, e.g.
module purge; module load nlpl-pytorch/1.3.0/3.7 module --terse list | grep -i python Python/3.7.2-GCCcore-8.2.0
Once the virtual environment is functional, it can be combined with NLPL modules, e.g.
module load nlpl-tensorflow/1.15.0/3.7 nlpl-horovod/0.18.2/3.7 . foo/bin/activate