Infrastructure/software/tensorflow
Background
TensorFlow is one of the most widely used Deep Learning frameworks in NLP (in mid-2018, at least), with corporate support from Google.
Usage on Abel
The module nlpl-tensorflow provides a TensorFlow installation in a Python 3.5 virtual environment. Besides TensorFlow and its dependencies (e.g. NumPy), the virtual environment includes a selection of popular add-on packages, e.g. SciKit-Learn, the Python Data Analysis Library (Pandas), GenSim, and Keras. This installation should support both cpu and gpu nodes on Abel.
module purge module use -a /projects/nlpl/software/modulefiles module load nlpl-tensorflow
There is a short sample program that test availability of cpu vs. gpu computing devices.
python /projects/nlpl/software/tensorflow/1.11/test.py
qlogin --account=nn9447k --time=1:00:00 --partition=accel --gres=gpu:1 module load nlpl-tensorflow python /projects/nlpl/software/tensorflow/1.11/test.py
Installation on Abel
module purge module load gcc/4.9.2 cuda/9.0 module load python3/3.5.0
cd /projects/nlpl/software mkdir tensorflow virtualenv tensorflow/1.11
First things first: Enable use of our custom (more modern) GNU C Library installation, by wrapping the basic python binary:
mv /projects/nlpl/software/tensorflow/1.11/bin/{,.}python3.5 sed 's@pytorch/0.4.1@tensorflow/1.11@' \ /projects/nlpl/software/pytorch/0.4.1/bin/python3.5 \ > /projects/nlpl/software/tensorflow/1.11/bin/python3.5 chmod 755 /projects/nlpl/software/tensorflow/1.11/bin/python3.5
Next, create a module definition, in this case /projects/nlpl/software/modulefiles/nlpl-tensorflow/1.11.
module load nlpl-tensorflow/1.11 pip install --upgrade pip pip install --upgrade $(pip list | tail -n +3 | awk '{print $1}') pip install --upgrade -r /projects/nlpl/software/tensorflow/1.11/modules.txt
qlogin --account=nn9447k --time=1:00:00 --partition=accel --gres=gpu:1 cp -av /usr/lib64/libcuda.so* /usr/lib64/libnvidia* \ /projects/nlpl/software/tensorflow/1.11/lib module purge module use -a /projects/nlpl/software/modulefiles module load nlpl-tensorflow pip install --upgrade tensorflow-gpu