Difference between revisions of "Translation/opennmt-py"

From Nordic Language Processing Laboratory
Jump to: navigation, search
(Created page with "= Usage on Abel = The module <tt>nlpl-opennmt-py</tt> provides an OpenNMT-py installation in a Python 3.5 virtual environment, building on the NLPL-maintained [http://wiki.nl...")
 
(→‎Installation on Taito)
 
(2 intermediate revisions by the same user not shown)
Line 18: Line 18:
 
module load nlpl-opennmt-py
 
module load nlpl-opennmt-py
 
</pre>
 
</pre>
 +
 +
The ‘wrapper’ scripts for common OpenNMT-py tasks (e.g. <tt>preprocess.py</tt>,
 +
<tt>train.py</tt>, and <tt>translate.py</tt> are installed into a <tt>scripts/</tt>
 +
subdirectory of the module, i.e. below the path
 +
<tt>/projects/nlpl/software/opennmt-py/0.2.1/scripts/</tt>.
  
 
= Available Versions =
 
= Available Versions =
Line 55: Line 60:
 
cp preprocess.py server.py train.py translate.py.py \
 
cp preprocess.py server.py train.py translate.py.py \
 
   /projects/nlpl/software/opennmt-py/0.2.1/scripts
 
   /projects/nlpl/software/opennmt-py/0.2.1/scripts
 +
</pre>
 +
 +
 +
= Installation on Taito =
 +
 +
<pre>
 +
module purge
 +
module load nlpl-pytorch
 +
cd /proj/nlpl/software
 +
svn co http://svn.nlpl.eu/software/opennmt-py
 +
virtualenv /proj/nlpl/software/opennmt-py/0.2.1
 +
</pre>
 +
 +
Next, the <tt>python</tt>, <tt>python3</tt>, and <tt>python3.5</tt> files
 +
(the first two soft links, the third an actual binary) had to be frobbed to
 +
avail themselves of the custom
 +
[http://wiki.nlpl.eu/index.php/Infrastructure/software/glibc NLPL C Library].
 +
 +
<pre>
 +
cd /proj/nlpl/software/opennmt-py/0.2.1/bin
 +
rm python python3 python3.5
 +
cp /proj/nlpl/software/glibc/2.18/wrapper python3.5
 +
ln -s python3.5 python3
 +
ln -s python3.5 python
 +
</pre>
 +
 +
<pre>
 +
mkdir /proj/nlpl/software/modulefiles/opennmt-py
 +
grep -v "cuda/9.0" /proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua \
 +
  | sed -e 's,pytorch,opennmt-py,' -e 's,0.4.1,0.2.1,' \
 +
    -e 's,python-env/3.5.3,nlpl-pytorch/0.4.1,' \
 +
  > /proj/nlpl/software/modulefiles/nlpl-opennmt-py/0.2.1.lua
 +
</pre>
 +
 +
<pre>
 +
module purge
 +
module load nlpl-opennmt-py
 +
wget https://github.com/OpenNMT/OpenNMT-py/archive/0.2.1.tar.gz
 +
tar zpSxvf 0.2.1.tar.gz
 +
cd OpenNMT-py-0.2.1
 +
pip install -r requirements.txt
 +
python setup.py install
 
</pre>
 
</pre>

Latest revision as of 19:54, 26 September 2018

Usage on Abel

The module nlpl-opennmt-py provides an OpenNMT-py installation in a Python 3.5 virtual environment, building on the NLPL-maintained PyTorch installation. Besides OpenNMT-py and its dependencies (e.g. PyTorch), the virtual environment includes a selection of popular add-on packages, e.g. NumPy, SciKit-Learn, the Python Data Analysis Library (Pandas), GenSim, and Keras. This installation should support both cpu and gpu nodes on Abel. 

module purge
module use -a /projects/nlpl/software/modulefiles
module load nlpl-opennmt-py

The ‘wrapper’ scripts for common OpenNMT-py tasks (e.g. preprocess.py, train.py, and translate.py are installed into a scripts/ subdirectory of the module, i.e. below the path /projects/nlpl/software/opennmt-py/0.2.1/scripts/.

Available Versions

As of September 2018, OpenNMT-py 0.2.1 is available (and, thus, the default version for this module).

Installation on Abel

module load nlpl-pytorch
mkdir /projects/nlpl/software/opennmt-py/
virtualenv /projects/nlpl/software/opennmt-py/0.2.1

Next, the python, python3, and python3.5 files (the first two soft links, the third an actual binary) had to be frobbed to avail themselves of the custom NLPL C Library. 

mkdir /projects/nlpl/software/modulefiles/opennmt-py
sed -e 's,pytorch,opennmt-py,g' \
  -e 's,PyTorch 0.4.1,OpenNMT 0.2.1 (PyTorch 0.4.1),' \
  -e 's,python3/3.5.0,nlpl-pytorch/0.4.1,' \
  /projects/nlpl/software/modulefiles/nlpl-pytorch/0.4.1 \
  > /projects/nlpl/software/modulefiles/opennmt-py/0.2.1

module purge
module load nlpl-opennmt-py
wget https://github.com/OpenNMT/OpenNMT-py/archive/0.2.1.tar.gz
tar zpSxvf 0.2.1.tar.gz
cd OpenNMT-py-0.2.1
python setup.py install
mkdir /projects/nlpl/software/opennmt-py/0.2.1/scripts
cp preprocess.py server.py train.py translate.py.py \
  /projects/nlpl/software/opennmt-py/0.2.1/scripts


Installation on Taito

module purge
module load nlpl-pytorch
cd /proj/nlpl/software
svn co http://svn.nlpl.eu/software/opennmt-py
virtualenv /proj/nlpl/software/opennmt-py/0.2.1

Next, the python, python3, and python3.5 files (the first two soft links, the third an actual binary) had to be frobbed to avail themselves of the custom NLPL C Library. 

cd /proj/nlpl/software/opennmt-py/0.2.1/bin
rm python python3 python3.5
cp /proj/nlpl/software/glibc/2.18/wrapper python3.5
ln -s python3.5 python3
ln -s python3.5 python

mkdir /proj/nlpl/software/modulefiles/opennmt-py
grep -v "cuda/9.0" /proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua \
  | sed -e 's,pytorch,opennmt-py,' -e 's,0.4.1,0.2.1,' \
    -e 's,python-env/3.5.3,nlpl-pytorch/0.4.1,' \
  > /proj/nlpl/software/modulefiles/nlpl-opennmt-py/0.2.1.lua

module purge
module load nlpl-opennmt-py
wget https://github.com/OpenNMT/OpenNMT-py/archive/0.2.1.tar.gz
tar zpSxvf 0.2.1.tar.gz
cd OpenNMT-py-0.2.1
pip install -r requirements.txt
python setup.py install
Retrieved from "http://wiki.nlpl.eu/index.php?title=Translation/opennmt-py&oldid=338"