Infrastructure/software/pytorch: Difference between revisions

From Nordic Language Processing Laboratory
Jump to navigation Jump to search
Line 7: Line 7:
of activity.
of activity.


= Usage on Abel =
= Usage =


The module <tt>nlpl-pytorch</tt> provides a PyTorch installation
The module <tt>nlpl-pytorch</tt> provides a PyTorch installation
in a Python 3.5 virtual environment.
in a Python 3.5 virtual environment.
Some [http://wiki.nlpl.eu/index.php/Infrastructure/software/python general background]
on ‘mixing and matching’ of NLPL Python modules is discussed as a
[http://wiki.nlpl.eu/index.php/Infrastructure/software/python separate page].
Besides PyTorch and its dependencies (e.g. NumPy), the virtual
Besides PyTorch and its dependencies (e.g. NumPy), the virtual
environment includes a selection of popular add-on packages, e.g.
environment includes a selection of popular add-on packages, e.g.
[http://scikit-learn.org/stable/documentation.html SciKit-Learn],
[http://scikit-learn.org/stable/documentation.html SciKit-Learn],
the [https://pandas.pydata.org/ Python Data Analysis Library] (Pandas),
the [https://pandas.pydata.org/ Python Data Analysis Library] (Pandas),
[https://radimrehurek.com/gensim/ GenSim],
and [https://keras.io/ Keras].
and [https://keras.io/ Keras].
This installation should support both cpu and gpu nodes on Abel.
This installation should support both cpu and gpu nodes on Abel and Taito.


<pre>
<pre>
module purge
module purge
module use -a /projects/nlpl/software/modulefiles
module use -a /proj*/nlpl/software/modulefiles
module load nlpl-pytorch
module load nlpl-pytorch
</pre>
</pre>
Line 29: Line 31:


<pre>
<pre>
python /projects/nlpl/software/pytorch/0.4.1/test.py
python /proj*/nlpl/software/pytorch/0.4.1/test.py
</pre>
</pre>



Revision as of 14:21, 2 October 2018

Background

In mid-2018 at least, PyTorch is one of the widely used programming frameworks for Deep Learning (with corporate backing by facebook); in and of itself PyTorch is not NLP-specific, but it is a critical dependency for several of the NLPL strands of activity.

Usage

The module nlpl-pytorch provides a PyTorch installation in a Python 3.5 virtual environment. Some general background on ‘mixing and matching’ of NLPL Python modules is discussed as a separate page. Besides PyTorch and its dependencies (e.g. NumPy), the virtual environment includes a selection of popular add-on packages, e.g. SciKit-Learn, the Python Data Analysis Library (Pandas), and Keras. This installation should support both cpu and gpu nodes on Abel and Taito.

module purge
module use -a /proj*/nlpl/software/modulefiles
module load nlpl-pytorch

There is a short sample program that test availability of cpu vs. gpu computing devices.

python /proj*/nlpl/software/pytorch/0.4.1/test.py

Available Versions

On Abel, as of September 2018, PyTorch 0.4.1 is available.

Installation on Abel

module purge
module load cuda/8.0
module load python3/3.5.0
cd /projects/nlpl/software
mkdir pytorch
virtualenv pytorch/0.4.1

Next, we need to create a module definition, in this case /projects/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.

module load nlpl-pytorch/0.4.1
pip install --upgrade pip
pip install --upgrade numpy pillow six
pip install torch torchvision
wget -O /tmp/requirements.txt \
  https://raw.githubusercontent.com/OpenNMT/OpenNMT-py/master/requirements.txt
pip install -r /tmp/requirements.txt

Installation on Taito

module purge
module load cuda/9.0
module load python-env/3.5.3
cd /proj/nlpl/software
svn co http://svn.nlpl.eu/software/pytorch
virtualenv pytorch/0.4.1

Next, we need to create a module definition, in this case /proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua.

module load nlpl-pytorch/0.4.1
pip install --upgrade pip
pip install --upgrade -r /proj/nlpl/software/pytorch/0.4.1/modules.txt