Difference between revisions of "Corpora/OPUS"

From Nordic Language Processing Laboratory
Jump to: navigation, search
(NLPL Users)
(http://opus.nlpl.eu)
 
(One intermediate revision by the same user not shown)
Line 2: Line 2:
 
== http://opus.nlpl.eu ==
 
== http://opus.nlpl.eu ==
  
OPUS is a collection of open parallel corpora in many languages. It provides bilingually aligned data sets, interfaces, tools and more. The data sets are available in various common formats and are provided for download and for use within the NLPL infrastructure. The service is hosted at CSC in Finland and the core of the data is also available from sigma2 on abel. Tools for processing the data are accessible from puhti and more detailed information can be found on the [http://opus.nlpl.eu/trac OPUS Wiki]:
+
OPUS is a collection of open parallel corpora in many languages. It provides bilingually aligned data sets, interfaces, tools and more. The data sets are available in various common formats and are provided for download and for use within the NLPL infrastructure. The service is hosted at CSC in Finland and the core of the data is also available from sigma2.
  
* Information for [http://opus.nlpl.eu/trac/wiki/NLPL NLPL Users]
+
For instructions on how to access the data and use the tools, check:
 +
* Information for [[#NLPL Users|NLPL Users]]
 +
 
 +
More detailed information can be found on the [http://opus.nlpl.eu/trac OPUS Wiki]:
 
* Information about the [http://opus.nlpl.eu/trac#WebAPI OPUS API] for finding resources
 
* Information about the [http://opus.nlpl.eu/trac#WebAPI OPUS API] for finding resources
 
* Information about [http://opus.nlpl.eu/trac/wiki/DataFormats data formats]
 
* Information about [http://opus.nlpl.eu/trac/wiki/DataFormats data formats]
Line 18: Line 21:
 
=== NLPL Users ===
 
=== NLPL Users ===
  
The OPUS corpus is now hosted at [[​https://www.csc.fi/ CSC]], the national scientific infrastructure provider of Finland and the resources are directly available for users of their services. The OPUS server runs in that environment but the data sets and tools are also directly available from the '''​puhti''' shell. The core data is also available on the Norwegian cluster ​'''saga''' provided by ​[[https://www.sigma2.no/ sigma2]].
+
The OPUS corpus is now hosted at [https://www.csc.fi/ CSC], the national scientific infrastructure provider of Finland and the resources are directly available for users of their services. The OPUS server runs in that environment but the data sets and tools are also directly available from the '''​puhti''' shell. The core data is also available on the Norwegian cluster ​'''saga''' provided by ​[https://www.sigma2.no/ sigma2].
  
 
If you have access to those systems then you will be able to access the data from the file system:
 
If you have access to those systems then you will be able to access the data from the file system:
 
on puhti: /projappl/nlpl/data/OPUS/
 
on saga: /projects/nlpl/data/OPUS/ (only raw XML data)
 
  
 
<pre>on puhti: /projappl/nlpl/data/OPUS/
 
<pre>on puhti: /projappl/nlpl/data/OPUS/
 
on saga: /projects/nlpl/data/OPUS/ (only raw XML data)</pre>
 
on saga: /projects/nlpl/data/OPUS/ (only raw XML data)</pre>
 
  
 
On both systems, you can also use tools that are packaged for working with the data (and other NLPL related activities). The basic tools for working with OPUS data can be loaded with the module nlpl-opus:
 
On both systems, you can also use tools that are packaged for working with the data (and other NLPL related activities). The basic tools for working with OPUS data can be loaded with the module nlpl-opus:
Line 42: Line 41:
  
 
With this, you will have access to essential tools that make it easier to read and process the data sets.
 
With this, you will have access to essential tools that make it easier to read and process the data sets.
 
=== Tools ===
 

Latest revision as of 12:52, 2 March 2020

http://opus.nlpl.eu

OPUS is a collection of open parallel corpora in many languages. It provides bilingually aligned data sets, interfaces, tools and more. The data sets are available in various common formats and are provided for download and for use within the NLPL infrastructure. The service is hosted at CSC in Finland and the core of the data is also available from sigma2.

For instructions on how to access the data and use the tools, check:

More detailed information can be found on the OPUS Wiki:

The on-line search interface is available from http://opus.nlpl.eu/bin/opuscqp.pl and the word-alignment-based lexicon is accessible from http://opus.nlpl.eu/lex.php

Contact: Jörg Tiedemann via e-mail - firstname.lastname at helsinki.fi (first name without dots)


NLPL Users

The OPUS corpus is now hosted at CSC, the national scientific infrastructure provider of Finland and the resources are directly available for users of their services. The OPUS server runs in that environment but the data sets and tools are also directly available from the ​puhti shell. The core data is also available on the Norwegian cluster ​saga provided by ​sigma2.

If you have access to those systems then you will be able to access the data from the file system:

on puhti: /projappl/nlpl/data/OPUS/
on saga: /projects/nlpl/data/OPUS/ (only raw XML data)

On both systems, you can also use tools that are packaged for working with the data (and other NLPL related activities). The basic tools for working with OPUS data can be loaded with the module nlpl-opus:

  • Activate the NLPL module repository:
    module use -a /projappl/nlpl/software/modules/etc         # Puhti
    module use -a /cluster/shared/nlpl/software/modules/etc   # Saga
  • Load the OPUS module:
    module load nlpl-opus

With this, you will have access to essential tools that make it easier to read and process the data sets.