Translation/opennmt-py

Usage on Abel

The module nlpl-opennmt-py provides an OpenNMT-py installation in a Python 3.5 virtual environment, building on the NLPL-maintained PyTorch installation. Besides OpenNMT-py and its dependencies (e.g. PyTorch), the virtual environment includes a selection of popular add-on packages, e.g. NumPy, SciKit-Learn, the Python Data Analysis Library (Pandas), GenSim, and Keras. This installation should support both cpu and gpu nodes on Abel.

module purge
module use -a /projects/nlpl/software/modulefiles
module load nlpl-opennmt-py

The ‘wrapper’ scripts for common OpenNMT-py tasks (e.g. preprocess.py, train.py, and translate.py are installed into a scripts/ subdirectory of the module, i.e. below the path /projects/nlpl/software/opennmt-py/0.2.1/scripts/.

Available Versions

As of September 2018, OpenNMT-py 0.2.1 is available (and, thus, the default version for this module).

Installation on Abel

module load nlpl-pytorch
mkdir /projects/nlpl/software/opennmt-py/
virtualenv /projects/nlpl/software/opennmt-py/0.2.1

Next, the python, python3, and python3.5 files (the first two soft links, the third an actual binary) had to be frobbed to avail themselves of the custom NLPL C Library.

mkdir /projects/nlpl/software/modulefiles/opennmt-py
sed -e 's,pytorch,opennmt-py,g' \
  -e 's,PyTorch 0.4.1,OpenNMT 0.2.1 (PyTorch 0.4.1),' \
  -e 's,python3/3.5.0,nlpl-pytorch/0.4.1,' \
  /projects/nlpl/software/modulefiles/nlpl-pytorch/0.4.1 \
  > /projects/nlpl/software/modulefiles/opennmt-py/0.2.1

module purge
module load nlpl-opennmt-py
wget https://github.com/OpenNMT/OpenNMT-py/archive/0.2.1.tar.gz
tar zpSxvf 0.2.1.tar.gz
cd OpenNMT-py-0.2.1
python setup.py install
mkdir /projects/nlpl/software/opennmt-py/0.2.1/scripts
cp preprocess.py server.py train.py translate.py.py \
  /projects/nlpl/software/opennmt-py/0.2.1/scripts

Installation on Taito

module purge
module load nlpl-pytorch
cd /proj/nlpl/software
svn co http://svn.nlpl.eu/software/opennmt-py
virtualenv /proj/nlpl/software/opennmt-py/0.2.1

Next, the python, python3, and python3.5 files (the first two soft links, the third an actual binary) had to be frobbed to avail themselves of the custom NLPL C Library.

cd /proj/nlpl/software/opennmt-py/0.2.1/bin
rm python python3 python3.5
cp /proj/nlpl/software/glibc/2.18/wrapper python3.5
ln -s python3.5 python3
ln -s python3.5 python

mkdir /proj/nlpl/software/modulefiles/opennmt-py
grep -v "cuda/9.0" /proj/nlpl/software/modulefiles/nlpl-pytorch/0.4.1.lua \
  | sed -e 's,pytorch,opennmt-py,' -e 's,0.4.1,0.2.1,' \
    -e 's,python-env/3.5.3,nlpl-pytorch/0.4.1,' \
  > /proj/nlpl/software/modulefiles/nlpl-opennmt-py/0.2.1.lua

module purge
module load nlpl-opennmt-py
wget https://github.com/OpenNMT/OpenNMT-py/archive/0.2.1.tar.gz
tar zpSxvf 0.2.1.tar.gz
cd OpenNMT-py-0.2.1
pip install -r requirements.txt
python setup.py install