Translation/opennmt-py

From Nordic Language Processing Laboratory
(Difference between revisions)
Jump to: navigation, search
(Installation on Abel)
(Installation on Taito)
 
Line 77: Line 77:
 
avail themselves of the custom
 
avail themselves of the custom
 
[http://wiki.nlpl.eu/index.php/Infrastructure/software/glibc NLPL C Library].
 
[http://wiki.nlpl.eu/index.php/Infrastructure/software/glibc NLPL C Library].
 +
 
<pre>
 
<pre>
 
cd /proj/nlpl/software/opennmt-py/0.2.1/bin
 
cd /proj/nlpl/software/opennmt-py/0.2.1/bin
Line 83: Line 84:
 
ln -s python3.5 python3
 
ln -s python3.5 python3
 
ln -s python3.5 python
 
ln -s python3.5 python
</pre
+
</pre>
  
 
<pre>
 
<pre>
mkdir /projects/nlpl/software/modulefiles/opennmt-py
+
mkdir /proj/nlpl/software/modulefiles/opennmt-py
sed -e 's,pytorch,opennmt-py,g' \
+
grep -v "cuda/9.0" /proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua \
  -e 's,PyTorch 0.4.1,OpenNMT 0.2.1 (PyTorch 0.4.1),' \
+
  | sed -e 's,pytorch,opennmt-py,' -e 's,0.4.1,0.2.1,' \
  -e 's,python3/3.5.0,nlpl-pytorch/0.4.1,' \
+
    -e 's,python-env/3.5.3,nlpl-pytorch/0.4.1,' \
   /projects/nlpl/software/modulefiles/nlpl-pytorch/0.4.1 \
+
   > /proj/nlpl/software/modulefiles/nlpl-opennmt-py/0.2.1.lua
  > /projects/nlpl/software/modulefiles/opennmt-py/0.2.1
+
 
</pre>
 
</pre>
  
Line 100: Line 100:
 
tar zpSxvf 0.2.1.tar.gz
 
tar zpSxvf 0.2.1.tar.gz
 
cd OpenNMT-py-0.2.1
 
cd OpenNMT-py-0.2.1
 +
pip install -r requirements.txt
 
python setup.py install
 
python setup.py install
mkdir /projects/nlpl/software/opennmt-py/0.2.1/scripts
 
cp preprocess.py server.py train.py translate.py.py \
 
  /projects/nlpl/software/opennmt-py/0.2.1/scripts
 
 
</pre>
 
</pre>

Latest revision as of 21:54, 26 September 2018

Contents

[edit] Usage on Abel

The module nlpl-opennmt-py provides an OpenNMT-py installation in a Python 3.5 virtual environment, building on the NLPL-maintained PyTorch installation. Besides OpenNMT-py and its dependencies (e.g. PyTorch), the virtual environment includes a selection of popular add-on packages, e.g. NumPy, SciKit-Learn, the Python Data Analysis Library (Pandas), GenSim, and Keras. This installation should support both cpu and gpu nodes on Abel.

module purge
module use -a /projects/nlpl/software/modulefiles
module load nlpl-opennmt-py

The ‘wrapper’ scripts for common OpenNMT-py tasks (e.g. preprocess.py, train.py, and translate.py are installed into a scripts/ subdirectory of the module, i.e. below the path /projects/nlpl/software/opennmt-py/0.2.1/scripts/.

[edit] Available Versions

As of September 2018, OpenNMT-py 0.2.1 is available (and, thus, the default version for this module).

[edit] Installation on Abel

module load nlpl-pytorch
mkdir /projects/nlpl/software/opennmt-py/
virtualenv /projects/nlpl/software/opennmt-py/0.2.1

Next, the python, python3, and python3.5 files (the first two soft links, the third an actual binary) had to be frobbed to avail themselves of the custom NLPL C Library.

mkdir /projects/nlpl/software/modulefiles/opennmt-py
sed -e 's,pytorch,opennmt-py,g' \
  -e 's,PyTorch 0.4.1,OpenNMT 0.2.1 (PyTorch 0.4.1),' \
  -e 's,python3/3.5.0,nlpl-pytorch/0.4.1,' \
  /projects/nlpl/software/modulefiles/nlpl-pytorch/0.4.1 \
  > /projects/nlpl/software/modulefiles/opennmt-py/0.2.1
module purge
module load nlpl-opennmt-py
wget https://github.com/OpenNMT/OpenNMT-py/archive/0.2.1.tar.gz
tar zpSxvf 0.2.1.tar.gz
cd OpenNMT-py-0.2.1
python setup.py install
mkdir /projects/nlpl/software/opennmt-py/0.2.1/scripts
cp preprocess.py server.py train.py translate.py.py \
  /projects/nlpl/software/opennmt-py/0.2.1/scripts


[edit] Installation on Taito

module purge
module load nlpl-pytorch
cd /proj/nlpl/software
svn co http://svn.nlpl.eu/software/opennmt-py
virtualenv /proj/nlpl/software/opennmt-py/0.2.1

Next, the python, python3, and python3.5 files (the first two soft links, the third an actual binary) had to be frobbed to avail themselves of the custom NLPL C Library.

cd /proj/nlpl/software/opennmt-py/0.2.1/bin
rm python python3 python3.5
cp /proj/nlpl/software/glibc/2.18/wrapper python3.5
ln -s python3.5 python3
ln -s python3.5 python
mkdir /proj/nlpl/software/modulefiles/opennmt-py
grep -v "cuda/9.0" /proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua \
  | sed -e 's,pytorch,opennmt-py,' -e 's,0.4.1,0.2.1,' \
    -e 's,python-env/3.5.3,nlpl-pytorch/0.4.1,' \
  > /proj/nlpl/software/modulefiles/nlpl-opennmt-py/0.2.1.lua
module purge
module load nlpl-opennmt-py
wget https://github.com/OpenNMT/OpenNMT-py/archive/0.2.1.tar.gz
tar zpSxvf 0.2.1.tar.gz
cd OpenNMT-py-0.2.1
pip install -r requirements.txt
python setup.py install
Personal tools
Namespaces

Variants
Actions
Navigation
Tools