Difference between revisions of "Infrastructure/software/user"

From Nordic Language Processing Laboratory
Jump to: navigation, search
(Created page with "= Background = Even if NLPL strives to make available a comprehensive set of ready-to-run [http://wiki.nlpl.eu/index.php/Infrastructure/software/catalogue sofware modules], u...")
 
(Python Virtual Environments on Saga)
 
Line 11: Line 11:
 
This design requires that some care is taken when creating user-level
 
This design requires that some care is taken when creating user-level
 
virtual environments; most importantly, the correct ''base'' Python version
 
virtual environments; most importantly, the correct ''base'' Python version
needs to be used when creating and manipulating the virtual environment:
+
needs to be used when creating and manipulating the virtual environment.
 +
In late 2019, we are building a first collection of NLPL Python add-ons
 +
on top of Python 3.7.2:
  
 
<pre>
 
<pre>
Line 29: Line 31:
 
</pre>
 
</pre>
  
Once the virtual environment is fully loaded, it can be combined with NLPL
+
One way of detecting the right ''base'' Python module to use for the creation
 +
of virtual environments is to inspect which version is loaded as prerequisites
 +
to a specific NLPL module, e.g.
 +
<pre>
 +
module purge; module load nlpl-pytorch/1.3.0/3.7
 +
module --terse list | grep -i python
 +
Python/3.7.2-GCCcore-8.2.0
 +
</pre>
 +
 
 +
Once the virtual environment is functional, it can be combined with NLPL
 
modules, e.g.
 
modules, e.g.
  

Latest revision as of 14:26, 28 October 2019

Background

Even if NLPL strives to make available a comprehensive set of ready-to-run sofware modules, users will at times want to install their own add-on components.

Python Virtual Environments on Saga

NLPL software modules that provide add-on components to Python are implemented as stacked virtual environments. This design requires that some care is taken when creating user-level virtual environments; most importantly, the correct base Python version needs to be used when creating and manipulating the virtual environment. In late 2019, we are building a first collection of NLPL Python add-ons on top of Python 3.7.2:

module purge; module load Python/3.7.2-GCCcore-8.2.0
python3 -m virtualenv foo
. foo/bin/activate
python3 -m pip install telegram

It is recommend to use python3 -m virtualenv and the like instead of short-hand commands like virtualenv or pip, because this gives better control over exactly which Python interpreter is used. For the above, for example, the following must hold (in late 2019):

type -path python3
/cluster/software/Python/3.7.2-GCCcore-8.2.0/bin/python3

One way of detecting the right base Python module to use for the creation of virtual environments is to inspect which version is loaded as prerequisites to a specific NLPL module, e.g.

module purge; module load nlpl-pytorch/1.3.0/3.7
module --terse list | grep -i python
Python/3.7.2-GCCcore-8.2.0

Once the virtual environment is functional, it can be combined with NLPL modules, e.g.

module load nlpl-tensorflow/1.15.0/3.7 nlpl-horovod/0.18.2/3.7
. foo/bin/activate