Translation/home

Background

An experimentation environment for Statistical and Neural Machine Translations (SMT and NMT) is maintained for NLPL under the coordination of the University of Helsinki (UoH). Initially, the software and data are commissioned on the Finnish Taito supercluster.

Current status (1/2018):

• moses module: SMT pipeline (Moses + various word alignment tools) installed on Taito and Abel (Moses release 4.0)
• efmaral module: efmaral and eflomal word alignment tools installed on Taito and Abel
• Older versions of moses and efmaral modules (installed 7/2017) are still available on Taito
• MT datasets available under /proj[ects]/nlpl/data/translation:
  • IWSLT17 (0.6G)
  • WMT17 News Task (16G)
  • Preprocessed WMT17 News files used for the Helsinki submissions (5G)

Coming up (Goal: 2/2018):

• NMT toolkits
• Pretrained models (SMT, HNMT)

Using the Moses module

• Log into Taito or Abel
• Activate the NLPL module repository:

module use -a /proj/nlpl/software/modulefiles/       # Taito
module use -a /projects/nlpl/software/modulefiles/   # Abel

• Load the most recent version of the Moses module:

module load moses

• Start using Moses, e.g. using the tutorial at http://statmt.org/moses/
  
• The module contains the standard installation as described at http://www.statmt.org/moses/?n=Development.GetStarted :
  • cmph, irstlm, xmlprc
  • with-mm
  • max-kenlm-order 10
  • max-factors 7
  • SALM + filter-pt
• For word alignment, you can use GIZA++, Mgiza and fast_align. (The word alignment tools efmaral and eflomal are part of a separate module.) If you need to specify absolute paths in your scripts, you can find them on the help page of the module:

module help moses

Using the Efmaral module

• Log into Taito or Abel
• Activate the NLPL module repository:

module use -a /proj/nlpl/software/modulefiles/       # Taito
module use -a /projects/nlpl/software/modulefiles/   # Abel

• Load the most recent version of the Efmaral module:

module load efmaral

• You can use the align.py script directly:

align.py ...

• You can use the efmaral module inside a Python3 script:

python3
>>> import efmaral

• You can test the example given at https://github.com/robertostling/efmaral by changing to the installation directory:

cd $EFMARALPATH
python3 scripts/evaluate.py efmaral \
   3rdparty/data/test.eng.hin.wa \
   3rdparty/data/test.eng 3rdparty/data/test.hin \
   3rdparty/data/trial.eng 3rdparty/data/trial.hin

• The Efmaral module also contains eflomal. You can use the alignment scripts as follows:

align_eflomal.py ...

• You can also use the eflomal executable:

eflomal ...

• You can also use the eflomal module in a Python3 script:

python3
>>> import eflomal

• The atools executable (from fast_align) is also made available.

Contact: Yves Scherrer, University of Helsinki, firstname.lastname@helsinki.fi