Difference between revisions of "Infrastructure/software/pytorch"
(→Usage on Abel) |
(→Usage) |
||
| (7 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
= Background = | = Background = | ||
| − | PyTorch is one of the widely used programming frameworks for | + | In mid-2018 at least, PyTorch is one of the widely used programming frameworks for |
| − | Deep Learning; in and of itself | + | Deep Learning (with corporate backing by facebook); |
| + | in and of itself PyTorch is not NLP-specific, but | ||
it is a critical dependency for several of the NLPL strands | it is a critical dependency for several of the NLPL strands | ||
of activity. | of activity. | ||
| − | = Usage | + | = Usage = |
The module <tt>nlpl-pytorch</tt> provides a PyTorch installation | The module <tt>nlpl-pytorch</tt> provides a PyTorch installation | ||
in a Python 3.5 virtual environment. | in a Python 3.5 virtual environment. | ||
| + | Some [http://wiki.nlpl.eu/index.php/Infrastructure/software/python general background] | ||
| + | on ‘mixing and matching’ of NLPL Python modules is discussed as a | ||
| + | [http://wiki.nlpl.eu/index.php/Infrastructure/software/python separate page]. | ||
Besides PyTorch and its dependencies (e.g. NumPy), the virtual | Besides PyTorch and its dependencies (e.g. NumPy), the virtual | ||
environment includes a selection of popular add-on packages, e.g. | environment includes a selection of popular add-on packages, e.g. | ||
[http://scikit-learn.org/stable/documentation.html SciKit-Learn], | [http://scikit-learn.org/stable/documentation.html SciKit-Learn], | ||
the [https://pandas.pydata.org/ Python Data Analysis Library] (Pandas), | the [https://pandas.pydata.org/ Python Data Analysis Library] (Pandas), | ||
| − | |||
and [https://keras.io/ Keras]. | and [https://keras.io/ Keras]. | ||
| − | This installation should support both cpu and gpu nodes on Abel. | + | This installation should support both cpu and gpu nodes on Abel and Taito. |
<pre> | <pre> | ||
| − | + | module use -a /proj*/nlpl/software/modulefiles | |
| − | module use -a / | ||
module load nlpl-pytorch | module load nlpl-pytorch | ||
</pre> | </pre> | ||
| + | |||
| + | There is a short sample program that tests availability of cpu vs. | ||
| + | gpu computing devices. | ||
| + | |||
| + | <pre> | ||
| + | python /proj*/nlpl/software/pytorch/0.4.1/test.py | ||
| + | </pre> | ||
| + | |||
| + | = Available Versions = | ||
| + | |||
| + | On Abel and Taito, as of September 2018, PyTorch 0.4.1 is available and the default version | ||
| + | of the <tt>nlpl-pytorch</tt> module. | ||
= Installation on Abel = | = Installation on Abel = | ||
| Line 53: | Line 67: | ||
https://raw.githubusercontent.com/OpenNMT/OpenNMT-py/master/requirements.txt | https://raw.githubusercontent.com/OpenNMT/OpenNMT-py/master/requirements.txt | ||
pip install -r /tmp/requirements.txt | pip install -r /tmp/requirements.txt | ||
| + | </pre> | ||
| + | |||
| + | = Installation on Taito = | ||
| + | |||
| + | <pre> | ||
| + | module purge | ||
| + | module load cuda/9.0 | ||
| + | module load python-env/3.5.3 | ||
| + | </pre> | ||
| + | |||
| + | <pre> | ||
| + | cd /proj/nlpl/software | ||
| + | svn co http://svn.nlpl.eu/software/pytorch | ||
| + | virtualenv pytorch/0.4.1 | ||
| + | </pre> | ||
| + | |||
| + | Next, we need to create a module definition, in this case | ||
| + | <tt>/proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua</tt>. | ||
| + | |||
| + | <pre> | ||
| + | module load nlpl-pytorch/0.4.1 | ||
| + | pip install --upgrade pip | ||
| + | pip install --upgrade -r /proj/nlpl/software/pytorch/0.4.1/modules.txt | ||
</pre> | </pre> | ||
Latest revision as of 12:33, 14 November 2018
Background
In mid-2018 at least, PyTorch is one of the widely used programming frameworks for Deep Learning (with corporate backing by facebook); in and of itself PyTorch is not NLP-specific, but it is a critical dependency for several of the NLPL strands of activity.
Usage
The module nlpl-pytorch provides a PyTorch installation in a Python 3.5 virtual environment. Some general background on ‘mixing and matching’ of NLPL Python modules is discussed as a separate page. Besides PyTorch and its dependencies (e.g. NumPy), the virtual environment includes a selection of popular add-on packages, e.g. SciKit-Learn, the Python Data Analysis Library (Pandas), and Keras. This installation should support both cpu and gpu nodes on Abel and Taito.
module use -a /proj*/nlpl/software/modulefiles module load nlpl-pytorch
There is a short sample program that tests availability of cpu vs. gpu computing devices.
python /proj*/nlpl/software/pytorch/0.4.1/test.py
Available Versions
On Abel and Taito, as of September 2018, PyTorch 0.4.1 is available and the default version of the nlpl-pytorch module.
Installation on Abel
module purge module load cuda/8.0 module load python3/3.5.0
cd /projects/nlpl/software mkdir pytorch virtualenv pytorch/0.4.1
Next, we need to create a module definition, in this case /projects/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.
module load nlpl-pytorch/0.4.1 pip install --upgrade pip pip install --upgrade numpy pillow six pip install torch torchvision
wget -O /tmp/requirements.txt \ https://raw.githubusercontent.com/OpenNMT/OpenNMT-py/master/requirements.txt pip install -r /tmp/requirements.txt
Installation on Taito
module purge module load cuda/9.0 module load python-env/3.5.3
cd /proj/nlpl/software svn co http://svn.nlpl.eu/software/pytorch virtualenv pytorch/0.4.1
Next, we need to create a module definition, in this case /proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua.
module load nlpl-pytorch/0.4.1 pip install --upgrade pip pip install --upgrade -r /proj/nlpl/software/pytorch/0.4.1/modules.txt
