Infrastructure/software/tensorflow

From Nordic Language Processing Laboratory
(Difference between revisions)
Jump to: navigation, search
(Installation on Taito)
(Installation on Taito)
Line 115: Line 115:
 
pip install --upgrade pip
 
pip install --upgrade pip
 
pip install --upgrade $(pip list | tail -n +3 | awk '{print $1}')
 
pip install --upgrade $(pip list | tail -n +3 | awk '{print $1}')
pip install --upgrade -r /projects/nlpl/software/tensorflow/1.11/modules.txt
+
pip install --upgrade -r /proj/nlpl/software/tensorflow/1.11/modules.txt
 
</pre>
 
</pre>
  
Line 126: Line 126:
 
module load nlpl-tensorflow
 
module load nlpl-tensorflow
 
pip install --upgrade tensorflow-gpu
 
pip install --upgrade tensorflow-gpu
 +
srun -n 1 -p gputest --gres=gpu:k80:1 --mem 1G -t 15 \
 +
  python /proj/nlpl/software/tensorflow/1.11/test.py
 
</pre>
 
</pre>

Revision as of 00:24, 30 September 2018

Contents

Background

TensorFlow is one of the most widely used Deep Learning frameworks in NLP (in mid-2018, at least), with corporate support from Google.

Usage on Abel

The module nlpl-tensorflow provides a TensorFlow installation in a Python 3.5 virtual environment. Besides TensorFlow and its dependencies (e.g. NumPy), the virtual environment includes a selection of popular add-on packages, e.g. SciKit-Learn, the Python Data Analysis Library (Pandas), GenSim, and Keras. This installation should support both cpu and gpu nodes on Abel.

module purge
module use -a /projects/nlpl/software/modulefiles
module load nlpl-tensorflow

There is a short sample program that test availability of cpu vs. gpu computing devices.

python /projects/nlpl/software/tensorflow/1.11/test.py
qlogin --account=nn9447k --time=1:00:00 --partition=accel --gres=gpu:1
module load nlpl-tensorflow
python /projects/nlpl/software/tensorflow/1.11/test.py

Available Versions

As of September 2018, TensorFlow 1.11 is available (and, thus, the default version for this module). An older installation of TensorFlow 1.0.1 is provided by USIT (the operators of Abel); this installation is ‘containerized’, however, i.e. is not easily interoperable with other software modules, and it does not work transparently on both cpu and gpu nodes.

Installation on Abel

module purge
module load gcc/4.9.2 cuda/9.0
module load python3/3.5.0
cd /projects/nlpl/software
mkdir tensorflow
virtualenv tensorflow/1.11

First things first: Enable use of our custom (more modern) GNU C Library installation, by wrapping the basic python binary:

mv /projects/nlpl/software/tensorflow/1.11/bin/{,.}python3.5
sed 's@pytorch/0.4.1@tensorflow/1.11@' \
  /projects/nlpl/software/pytorch/0.4.1/bin/python3.5 \
  > /projects/nlpl/software/tensorflow/1.11/bin/python3.5
chmod 755 /projects/nlpl/software/tensorflow/1.11/bin/python3.5

Next, create a module definition, in this case /projects/nlpl/software/modulefiles/nlpl-tensorflow/1.11.

module load nlpl-tensorflow/1.11
pip install --upgrade pip
pip install --upgrade $(pip list | tail -n +3 | awk '{print $1}')
pip install --upgrade -r /projects/nlpl/software/tensorflow/1.11/modules.txt
qlogin --account=nn9447k --time=1:00:00 --partition=accel --gres=gpu:1
cp -av /usr/lib64/libcuda.so* /usr/lib64/libnvidia* \
  /projects/nlpl/software/tensorflow/1.11/lib
module purge
module use -a /projects/nlpl/software/modulefiles
module load nlpl-tensorflow
pip install --upgrade tensorflow-gpu

Installation on Taito

module purge
module load cuda-env/9.0
module load python-env/3.5.3
cd /proj/nlpl/software
svn co http://svn.nlpl.eu/software/tensorflow
virtualenv /proj/nlpl/software/tensorflow/1.11

First things first: Enable use of our custom (more modern) GNU C Library installation, by wrapping the basic python binary:

mv /proj/nlpl/software/tensorflow/1.11/bin/{,.}python3.5
cp /proj/nlpl/software/glibc/2.18/wrapper \
  /proj/nlpl/software/tensorflow/1.11/bin/python3.5

Next, create a module definition, in this case /proj/nlpl/software/modulefiles/nlpl-tensorflow/1.11.lua.

module load nlpl-tensorflow/1.11
pip install --upgrade pip
pip install --upgrade $(pip list | tail -n +3 | awk '{print $1}')
pip install --upgrade -r /proj/nlpl/software/tensorflow/1.11/modules.txt
ssh taito-gpu
cp -av /usr/lib64/libcuda.so* /usr/lib64/libnvidia* \
  /proj/nlpl/software/tensorflow/1.11/lib
module purge
module use -a /projects/nlpl/software/modulefiles
module load nlpl-tensorflow
pip install --upgrade tensorflow-gpu
srun -n 1 -p gputest --gres=gpu:k80:1 --mem 1G -t 15 \
  python /proj/nlpl/software/tensorflow/1.11/test.py
Personal tools
Namespaces

Variants
Actions
Navigation
Tools