Translation/home - Revision history

Yvessche at 11:54, 21 October 2022

2022-10-21T11:54:18Z

Yvessche at 11:53, 21 October 2022

2022-10-21T11:53:17Z

Drobac: /* Tools for processing parallel corpora (OPUS tools) */

2020-03-02T10:50:06Z

‎Tools for processing parallel corpora (OPUS tools)

Drobac: /* Available software and data */

2020-03-02T10:19:40Z

‎Available software and data

Drobac: /* Using the OPUS Tools module */

2020-03-02T10:12:51Z

‎Using the OPUS Tools module

Drobac: /* Using the OPUS Tools module */

2020-03-02T10:12:26Z

‎Using the OPUS Tools module

Drobac: /* Using the Efmaral module */

2020-03-02T10:09:19Z

‎Using the Efmaral module

Drobac: /* Tools for processing parallel corpora (OPUS tools) */

2020-03-02T10:07:42Z

‎Tools for processing parallel corpora (OPUS tools)

Drobac: /* Tools for processing parallel corpora (OPUS tools) */

2020-03-02T10:07:18Z

‎Tools for processing parallel corpora (OPUS tools)

Drobac at 10:06, 2 March 2020

2020-03-02T10:06:50Z

@@ Line 85: / Line 85: @@
 </ul></li>
 <li>For word alignment, you can use GIZA++, Mgiza and fast_align. (The word alignment tools efmaral and eflomal are part of a [[#Using the Efmaral module|separate module]].)<br/>If you need to specify absolute paths in your scripts, you can find them on the help page of the module:
-<pre>module help nlpl-moses/4.0-a89691f</pre>
+<pre>module help nlpl-moses</pre>
 </li>
 </ul>

@@ Line 2: / Line 2: @@
 [[Translation/taito_abel|Translation activity on the Taito and Abel servers (outdated)]]
 An experimentation environment for Statistical and Neural Machine Translations (SMT and NMT)
@@ Line 13: / Line 11: @@
 === Statistical machine translation and word alignment ===
-* The '''Moses''' SMT pipeline with word alignment tools GIZA++, MGIZA, fast_align, with SALM (release 4.0) is installed on Puhti and Saga: <code>nlpl-moses/4.0-a89691f</code> ([[#Using the Moses module|usage notes below]])
+* The '''Moses''' SMT pipeline with word alignment tools GIZA++, MGIZA, fast_align, with SALM (release 4.0) is installed on Puhti and Saga: <code>nlpl-moses/4.0-a89691f</code> ([[#Using the Moses module|usage notes below]]). Note: the most recent version on Puhti (as of Oct 2022) is <code>nlpl-moses/4.0.1-3990724</code>.
-* The word alignment tools '''efmaral and eflomal''' are installed on Puhti and Saga in the nlpl-efmaral module: <code>nlpl-efmaral/0.1_20191218</code> ([[#Using the Efmaral module|usage notes below]])
+* The word alignment tools '''efmaral and eflomal''' are installed on Puhti and Saga in the nlpl-efmaral module: <code>nlpl-efmaral/0.1_20191218</code> ([[#Using the Efmaral module|usage notes below]]). Note: the most recent version on Puhti (as of Oct 2022) is <code>nlpl-efmaral/1.0.1_20221015</code>.
 === Neural machine translation ===
@@ Line 20: / Line 18: @@
 * '''Marian-NMT''' is installed on Puhti and Saga as <code>nlpl-marian-nmt/1.8.0-eba7aed</code>. [[#Using the Marian-NMT module|Usage notes below.]]
 * '''OpenNMT-py''' is installed on Saga using NLPL-internal Pytorch: <code>nlpl-opennmt-py/1.0.0rc2/3.7</code>.
-* '''OpenNMT-py''' is installed on Puhti using system-wide Pytorch: <code>nlpl-opennmt-py/nlpl-opennmt-py/1.0.0</code>.
+* '''OpenNMT-py''' is installed on Puhti using system-wide Pytorch: <code>nlpl-opennmt-py/2.3.0</code>.
 === General scripts for machine translation ===
-* The '''nlpl-mttools''' module provides a series of preprocessing and evaluation scripts useful for any kind of machine translation research, independently of the toolkit. It is installed on Puhti and Saga: <code>nlpl-mttools/20191218</code>. See [[Translation/mttools|the mttools page]] for further details.
+* The '''nlpl-mttools''' module provides a series of preprocessing and evaluation scripts useful for any kind of machine translation research, independently of the toolkit. It is installed on Puhti and Saga: <code>nlpl-mttools/20191218</code>. See [[Translation/mttools|the mttools page]] for further details. Note: the most recent version on Puhti (as of Oct 2022) is <code>nlpl-mttools/20221015</code>.
 === Tools for processing parallel corpora (OPUS tools) ===
@@ Line 74: / Line 72: @@
 module use -a /cluster/shared/nlpl/software/modules/etc   # Saga</pre>
 </li>
-<li>Load the Moses module:
+<li>Load the most recent version of the Moses module:
-<pre>module load nlpl-moses/4.0-a89691f</pre>
+<pre>module load nlpl-moses</pre>
 </li>
 <li>Start using Moses, e.g. using the tutorial at http://statmt.org/moses/</li>
@@ Line 98: / Line 96: @@
 module use -a /cluster/shared/nlpl/software/modules/etc   # Saga</pre>
 </li>
-<li>Load the Efmaral module:
+<li>Load the most recent version of the Efmaral module:
 <pre>
-module load nlpl-efmaral/0.1_20191218
+module load nlpl-efmaral
 </pre>
 </li>

@@ Line 29: / Line 29: @@
 * The bundle of '''OPUS tools''' is installed on Puhti and Saga in the <code>nlpl-opus</code> module. [[#Using the OPUS Tools module|Usage notes below.]]
 * '''Uplug''' is installed in the <code>nlpl-uplug</code> module.
 * '''Corpus Work Bench''' is installed in the <code>nlpl-cwb</code> module.

@@ Line 27: / Line 27: @@
 === Tools for processing parallel corpora (OPUS tools) ===
-* The bundle of '''OPUS tools''' is installed on Puhti and Saga in the <code>nlpl-opus</code> module.
+* The bundle of '''OPUS tools''' is installed on Puhti and Saga in the <code>nlpl-opus</code> module. [[#Using the OPUS Tools module|Usage notes below.]]
 * '''Uplug''' is installed in the <code>nlpl-uplug</code> module.
 * '''Corpus Work Bench''' is installed in the <code>nlpl-cwb</code> module.

@@ Line 141: / Line 141: @@
 </pre>
 </li>
-<li>You can also load Uplug, Udpipe and CWB modules:
+<li>You can also load CWB, Uplug and Udpipe modules:
 <pre>module load nlpl-cwb</pre>
 <pre>module nlpl-uplug</pre>

← Older revision		Revision as of 10:09, 2 March 2020
Line 91:		Line 91:

	= Using the Efmaral module =		= Using the Efmaral module =
		+
		+	<ul>
		+	<li>Activate the NLPL module repository:
		+	<pre>module use -a /projappl/nlpl/software/modules/etc # Puhti
		+	module use -a /cluster/shared/nlpl/software/modules/etc # Saga</pre>
		+	</li>
		+	<li>Load the Efmaral module:
		+	<pre>
		+	module load nlpl-efmaral/0.1_20191218
		+	</pre>
		+	</li>
		+	<li>You can use the align.py script directly:
		+	<pre>align.py ...</pre>
		+	</li>
		+	<li>You can use the efmaral module inside a Python3 script:
		+	<pre>python3
		+	>>> import efmaral</pre>
		+	</li>
		+	<li>You can test the example given at https://github.com/robertostling/efmaral by changing to the installation directory:
		+	<pre>cd $EFMARALPATH
		+	python3 scripts/evaluate.py efmaral \
		+	3rdparty/data/test.eng.hin.wa \
		+	3rdparty/data/test.eng 3rdparty/data/test.hin \
		+	3rdparty/data/trial.eng 3rdparty/data/trial.hin</pre>
		+	</li>
		+	<li>The Efmaral module also contains eflomal. You can use the alignment scripts as follows:
		+	<pre>align_eflomal.py ...</pre>
		+	</li>
		+	<li>You can also use the eflomal executable:
		+	<pre>eflomal ...</pre>
		+	</li>
		+	<li>You can also use the eflomal module in a Python3 script:
		+	<pre>python3
		+	>>> import eflomal</pre>
		+	</li>
		+	<li>The atools executable (from fast_align) is also made available.</li>
		+	</ul>
		+
		+	= Using the OPUS Tools module =

	<ul>		<ul>

← Older revision		Revision as of 10:06, 2 March 2020
Line 25:		Line 25:

	* The '''nlpl-mttools''' module provides a series of preprocessing and evaluation scripts useful for any kind of machine translation research, independently of the toolkit. It is installed on Puhti and Saga: <code>nlpl-mttools/20191218</code>. See [[Translation/mttools\|the mttools page]] for further details.		* The '''nlpl-mttools''' module provides a series of preprocessing and evaluation scripts useful for any kind of machine translation research, independently of the toolkit. It is installed on Puhti and Saga: <code>nlpl-mttools/20191218</code>. See [[Translation/mttools\|the mttools page]] for further details.
		+
		+	=== Tools for processing parallel corpora (OPUS tools) ===
		+	* The bundle of OPUS tools is installed on Puhti and Saga in the <code>nlpl-opus module</code>.
		+	* '''Uplug''' is installed in the <code>nlpl-uplug</code> module.
		+	* '''Corpus Work Bench''' is installed in the <code>nlpl-cwb</code> module.

	=== Datasets ===		=== Datasets ===