Translation/models

MT example scripts and pretrained models

The models and scripts are located at /proj*/nlpl/data/translation/pretrained-models/

wmt18_helsinki-enfi-moses

This directory contains training scripts and the resulting model files for a translation system:

• trained on WMT18 news data preprocessed by the University of Helsinki
• translating from English to Finnish
• using the Moses SMT toolkit.

The scripts (and the resulting model) are based on the Moses tutorial which has additional information. The goals of this example are twofold:

• to illustrate the use of the Moses tools and MT data as provided by the NLPL project for training new models,
• to provide a pre-trained, ready-to-use translation model.

The two use cases are described below.

Retrain a new model using the provided scripts

• Copy the `1_prepare.sh` to `6_test.sh` scripts to your own working directory.
• Adapt paths if necessary, e.g. if you want to use data for a different language pair or a different translation direction.
• Run the scripts one by one. The time and memory requirements in the SLURM scripts are tuned to usage on Taito in January 2019 and may have to be adapted. The scripts have not been tested on Abel, but should run with the necessary adaptations.
• The output of script 6 should be similar to the provided files `testdata.out.fi` and `evaluation.txt`. Minor differences can be expected due to the non-deterministic nature of MERT tuning.

Use the pre-trained model to translate unseen text

• Copy the `6_test.sh` script to your own working directory.
• Provide a tokenized and truecased test file (`1_prepare.sh` shows how to do that) or copy `testdata.en` to your working directory.
• Adapt the WORKDIR path in `6_test.sh` and run the script.
• The output of script 6 corresponds to the files `testdata.out.fi` and `evaluation.txt`. Note that evaluation will only work correctly if the test set is registered in the sacreBLEU database. This is typically the case for WMT and IWSLT test sets.

Yves Scherrer, January 2019

wmt18_helsinki-enfi-onmt

This directory contains training scripts and the resulting model files for a translation system:

• trained on WMT18 news data preprocessed by the University of Helsinki
• translating from English to Finnish
• using the OpenNMT-py toolkit.

The scripts (and the resulting model) are a slightly simplified version of the original Helsinki submissions. The goals of this example are twofold:

• to illustrate the use of the OpenNMT-py library and MT data as provided by the NLPL project for training new models,
• to provide a pre-trained, ready-to-use translation model.

The two use cases are described below.

Retrain a new model using the provided scripts

• Copy the `1_prepare.sh` to `4_test.sh` scripts to your own working directory.
• Adapt paths if necessary, e.g. if you want to use data for a different language pair or a different translation direction.
• Run the scripts one by one. The time and memory requirements in the SLURM scripts are tuned to usage on Taito in May 2019 and may have to be adapted.
• The scripts will not run out-of-the-box on Abel due to different installed versions of OpenNMT-py. The relevant module can be loaded on Abel with `module load nlpl-opennmt-py` (without the `-gpu` suffix).
• The output of script 4 should be similar to the provided files `testdata.out.fi` and `evaluation.txt`. Minor differences can be expected due to the non-deterministic nature of neural network training on GPU.

Use the pre-trained model to translate unseen text

• Copy the `4_test.sh` script to your own working directory.
• Provide a tokenized, truecased and BPE-encoded test file (`1_prepare.sh` shows how to do that) or copy `testdata.en` to your working directory.
• Adapt the WORKDIR path in `4_test.sh` and run the script.
• The output of script 4 corresponds to the files `testdata.out.fi` and `evaluation.txt`. Note that evaluation will only work correctly if the test set is registered in the sacreBLEU database. This is typically the case for WMT and IWSLT test sets.

Yves Scherrer, May 2019

wmt18-fien-moses

This directory contains training scripts and the resulting model files for a translation system:

• trained on the raw versions of the WMT18 news data
• translating from Finnish to English
• using the Moses SMT toolkit.

The scripts (and the resulting model) are based on the Moses tutorial which has additional information. The goals of this example are twofold:

• to illustrate the use of the preprocessing pipeline, the Moses tools and MT data as provided by the NLPL project for training new models,
• to provide a pre-trained, ready-to-use translation model.

The two use cases are described below.

Retrain a new model using the provided scripts

• Copy the `1_prepare.sh` to `6_test.sh` scripts to your own working directory.
• Adapt paths if necessary, e.g. if you want to use data for a different language pair or a different translation direction.
• Run the scripts one by one. The time and memory requirements in the SLURM scripts are tuned to usage on Taito in January 2019 and may have to be adapted. The scripts have not been tested on Abel, but should run with the necessary adaptations.
• The output of script 6 should be similar to the provided files `testdata.out.en` and `evaluation.txt`. Minor differences can be expected due to the non-deterministic nature of MERT tuning.

Use the pre-trained model to translate unseen text

• Copy the `6_test.sh` script to your own working directory.
• Provide a tokenized and truecased test file (`1_prepare.sh` shows how to do that) or copy `testdata.en` to your working directory.
• Adapt the WORKDIR path in `6_test.sh` and run the script.
• The output of script 6 corresponds to the files `testdata.out.en` and `evaluation.txt`. Note that evaluation will only work correctly if the test set is registered in the sacreBLEU database. This is typically the case for WMT and IWSLT test sets.

Yves Scherrer, January 2019

wmt18-fien-onmt

This directory contains training scripts and the resulting model files for a translation system:

• trained on the raw versions of the WMT18 news data
• translating from Finnish to English
• using the OpenNMT-py toolkit.

The scripts (and the resulting model) are a slightly simplified version of the original Helsinki submissions. The goals of this example are twofold:

• to illustrate the use of the preprocessing pipeline, the OpenNMT-py library and MT data as provided by the NLPL project for training new models,
• to provide a pre-trained, ready-to-use translation model.

The two use cases are described below.

Retrain a new model using the provided scripts

• Copy the `1_prepare.sh` to `4_test.sh` scripts to your own working directory.
• Adapt paths if necessary, e.g. if you want to use data for a different language pair or a different translation direction.
• Run the scripts one by one. The time and memory requirements in the SLURM scripts are tuned to usage on Taito in May 2019 and may have to be adapted.
• The scripts will not run out-of-the-box on Abel due to different installed versions of OpenNMT-py. The relevant module can be loaded on Abel with `module load nlpl-opennmt-py` (without the `-gpu` suffix).
• The output of script 6 should be similar to the provided files `testdata_out.en` and `evaluation.txt`. Minor differences can be expected due to the non-deterministic nature of MERT tuning.

Use the pre-trained model to translate unseen text

• Copy the `4_test.sh` script to your own working directory.
• Provide a tokenized and truecased test file (`1_prepare.sh` shows how to do that) or copy `testdata.en` to your working directory.
• Adapt the WORKDIR path in `4_test.sh` and run the script.
• The output of script 4 corresponds to the files `testdata_out.en` and `evaluation.txt`. Note that evaluation will only work correctly if the test set is registered in the sacreBLEU database. This is typically the case for WMT and IWSLT test sets.

Yves Scherrer, May 2019

opus-noen-moses

This directory contains training scripts and the resulting model files for a translation system:

• trained on sentence-aligned data from the OPUS collection
• translating from Norwegian to English
• using the Moses SMT toolkit.

The scripts (and the resulting model) are based on the Moses tutorial which has additional information. The goals of this example are twofold:

• to illustrate the use of the preprocessing pipeline, the Moses tools and the OPUS corpus collection as provided by the NLPL project for training new models,
• to provide a pre-trained, ready-to-use translation model for a "low-resource language" from an MT point of view, Norwegian.

The two use cases are described below.

Retrain a new model using the provided scripts

• Copy the `1_prepare.sh` to `6_test.sh` scripts to your own working directory.
• Adapt paths if necessary, e.g. if you want to use data for a different language pair or a different translation direction.
• Run the scripts one by one. The time and memory requirements in the SLURM scripts are tuned to usage on Taito in May 2019 and may have to be adapted. The scripts have not been tested on Abel, but should run with the necessary adaptations.
• The output of script 6 should be similar to the provided files `testdata_out.tok.en`, `testdata_out.en` and `evaluation.txt`. Minor differences can be expected due to the non-deterministic nature of MERT tuning.

Use the pre-trained model to translate unseen text

• Copy the `6_test.sh` script to your own working directory.
• Provide a tokenized and truecased test file (`1_prepare.sh` shows how to do that) or copy `testdata.true.no` to your working directory.
• Adapt the WORKDIR path in `6_test.sh` and run the script.
• The output of script 6 corresponds to the files `testdata_out.tok.en`, `testdata_out.en` and `evaluation.txt`.

Yves Scherrer, May 2019

opus-noen-onmt

This directory contains training scripts and the resulting model files for a translation system:

• trained on sentence-aligned data from the OPUS collection
• translating from Norwegian to English
• using the OpenNMT-py toolkit.

The goals of this example are twofold:

• to illustrate the use of the preprocessing pipeline, the OpenNMT-py library and the OPUS corpus collection as provided by the NLPL project for training new models,
• to provide a pre-trained, ready-to-use translation model for a "low-resource language" from an MT point of view, Norwegian.

The two use cases are described below.

Retrain a new model using the provided scripts

• Copy the `1_prepare.sh` to `4_test.sh` scripts to your own working directory.
• Adapt paths if necessary, e.g. if you want to use data for a different language pair or a different translation direction.
• Run the scripts one by one. The time and memory requirements in the SLURM scripts are tuned to usage on Taito in May 2019 and may have to be adapted.
• The scripts will not run out-of-the-box on Abel due to different installed versions of OpenNMT-py. The relevant module can be loaded on Abel with `module load nlpl-opennmt-py` (without the `-gpu` suffix).
• The output of script 4 should be similar to the provided files `testdata_out.en` and `evaluation.txt`. Minor differences can be expected due to the non-deterministic nature of MERT tuning.

Use the pre-trained model to translate unseen text

• Copy the `4_test.sh` script to your own working directory.
• Provide a tokenized and truecased test file (`1_prepare.sh` shows how to do that) or copy `testdata.en` to your working directory.
• Adapt the WORKDIR path in `4_test.sh` and run the script.
• The output of script 4 corresponds to the files `testdata_out.en` and `evaluation.txt`. Note that evaluation will only work correctly if the test set is registered in the sacreBLEU database. This is typically the case for WMT and IWSLT test sets.

Yves Scherrer, May 2019

iwslt18_helsinki-euen-marian

This directory contains training scripts and the resulting model files for a translation system:

• trained on data from the IWSLT18 low-resource translation task on Basque-to-English
• using the preprocessed and augmented datasets from the University of Helsinki submission
• with the Marian NMT toolkit.

The scripts (and the resulting model) correspond to a slightly simplified version of the original Helsinki submission. The goals of this example are twofold:

• to illustrate the use of the Marian library and the MT data sets as provided by the NLPL project for training new models,
• to provide a pre-trained, ready-to-use translation model.

The two use cases are described below.

Retrain a new model using the provided scripts

• Copy the `1_train.sh`, `2_test.sh`, `validate.sh` and `composeXML.py` scripts to your own working directory.
• Adapt paths if necessary.
• Run the script `1_train.sh`, then `2_test.sh`. The `validate.sh` script is automatically called during training and does not have to be run separately. The `composeXML.py` script is automatically called during testing and does not have to be r un separately. The time and memory requirements in the SLURM scripts are tuned to usage on Taito in January 2019 and may have to be adapted. Note that these scripts use the Marian version installed system-wide on Taito and may not run correctly on the earlier NLPL-installed Marian version available on Abel.
• The output of script 2 should be similar to the provided `test.out.en` and `test.out.en.xml` files. Minor differences can be expected due to the non-deterministic nature of neural network training on GPU.

Use the pre-trained model to translate unseen text

• Copy the `2_test.sh` and `composeXML.py` scripts to your own working directory.
• Provide a tokenized, truecased and BPE-encoded test file or copy `test.eu` to your working directory.
• Adapt the WORKDIR path in `2_test.sh` and run the script.
• The output of script 2 corresponds to the files `test.out.en` and `test.out.en.xml`.
• The result XML file is sent to the evaluation server. Uncomment this if you are not translating the official IWSLT 2018 test set.

Yves Scherrer, January 2019