Nordic Language Processing Laboratory - User contributions [en]

Home

2019-02-04T16:13:13Z

Figint: /* Resources */

The Nordic Language Processing Laboratory (NLPL) is a collaboration of
university research groups in Natural Language Processing (NLP) in Northern Europe.
Our vision is to implement a virtual laboratory for large-scale NLP research by
(a) creating new ways to enable data- and compute-intensive Natural Language Processing research by implementing a common software, data and service stack in multiple Nordic HPC centres,
(b) by pooling competencies within the user community and among expert support teams,
and (c) by enabling internationally competitive, data-intensive research and experimentation
on a scale that would be difficult to sustain on commodity computing resources.

= Activities =

<br/>
[[File:neic.png|center]]
<br/><br/>

As part of its ‘virtual laboratory’, NLPL prepares and maintains
[http://wiki.nlpl.eu/index.php/Infrastructure/software/catalogue software] and data infrastructures for
(A) [http://wiki.nlpl.eu/index.php/Infrastructure/home Collaboration and Software Management];
(B) [http://wiki.nlpl.eu/index.php/Translation/home Statistical and Neural Machine Translation];
(C) [http://wiki.nlpl.eu/index.php/Parsing/home Data-Driven Dependency Parsing];
(D) [http://wiki.nlpl.eu/index.php/Corpora/home Very Large Corpora];
(E) [http://wiki.nlpl.eu/index.php/Vectors/home Pre-Trained Word Embeddings];
(F) [http://wiki.nlpl.eu/index.php/Evaluation/home Automated Extrinsic Evaluation];
(G) [http://wiki.nlpl.eu/index.php/Corpora/OPUS Parallel Corpora and OPUS]; and
(H) [http://wiki.nlpl.eu/index.php/Community/home Community Formation and Outreach].
Please see the [http://wiki.nlpl.eu/index.php/Infrastructure/software/catalogue catalogue of available software]
and above links for information on how to gain access to and utilize the NLPL virtual laboratory.

= Resources =

Since mid-2017, NLPL has started to make available some of its resources and services to the public:

* [http://hdl.handle.net/11234/1-1989 90 billion tokens of ‘raw’ text] extracted from web data, covering the 45 languages in the 2017 UD Parsing Shared Task; [http://wiki.nlpl.eu/index.php/Corpora/home see here]
* The [http://corpora.nlpl.eu/engc3/ EngC3] corpus of some [http://corpora.nlpl.eu/engc3/ 130 billion tokens of clean English text] extraced from the Common Crawl;
* The [http://epe.nlpl.eu Extrinsic Parser Evaluation 2017] (EPE) Shared Task at the DepLing and IWPT 2017 conferences;
* A [http://vectors.nlpl.eu/repository repository of pre-trained word embeddings] on very large corpora and [http://vectors.nlpl.eu/explore on-line explorer] for these models;
* The [http://opus.nlpl.eu Open Parallel Corpus] (OPUS; now maintained as a dedicated service under the NLPL umbrella);
* An annual [http://wiki.nlpl.eu/index.php/Community/training winter school series] on machine learning and scientific programming for NLP research.

= Partners =

The NLPL consortium is comprised of Nordic research groups in NLP and
the national e-infrastructure providers of Finland and Norway:
Helsinki University (Finland), IT University Copenhagen (Denmark),
University of Copenhagen (Denmark), University of Oslo (Norway),
Turku University (Finland), and Uppsala University (Sweden) are the
academic partners.

Between 2017 and 2020, NLPL is supported by the [https://neic.nordforsk.org/ Nordic e-Infrastructure Collaboration]
(NeIC) and the national e-Infrastructure providers in Finland ([http://www.csc.fi CSC]) and Norway ([https://www.sigma2.no/ Sigma2]).

= Associates =

NLPL welcomes involvement of additional research groups in Language Technology in the Nordics, including the Baltic region, to make use of the virtual laboratory. The project has established an associate program where users can get access to NLPL resources.
Please email the contact address below to ask for access.
As part of your initial contact, please provide an indication of the
expected types of computing, software, and data to be used and the
anticipated group of users (including details on affiliation).

As of October 2018, the following research groups are NLPL associates:

* [https://clasp.gu.se/ Center for Linguistic Theory and Studies of Probability] at Gothenburg University (Sweden)
* [https://www.ling.su.se/english/nlp Section for Computational Linguistics] at Stockholm University (Sweden)
* [https://nlp.cs.ut.ee/ Natural Language Processing Research Group] at the University of Tartu (Estonia)

= Contact =

To email NLPL project management and its Steering Group, please use the address <code>contact</code><code>@</code><code>nlpl.eu</code>.
In mid-2017, the project welcomes expressions of interest from additional NLP research groups in Northern Europe.

For additional background and the archive of official project documents (including the work plan and Steering Group minutes), please
see the [https://wiki.neic.no/wiki/Nordic_language_processing_laboratory NLPL page on the NeIC wiki].

Corpora/home

2019-02-04T16:06:04Z

Figint: /* Many Languages: The CoNLL 2017 Text Collection (Turku) */

= Background =

NLPL creates and makes available various very large collections of textual data, for example drawing on Wikipedia and the Common Crawl.

These data resources are available from the connected infrastructure, below the project directory
<tt>/projects/nlpl/data/corpora/</tt> or <tt>/proj/nlpl/data/corpora/</tt> on Abel and Taito, respectively.
In early 2018, the NLPL corpora resources are not yet replicated across the two systems; please see
the summary table below to determine which resource is available on which system.

= Corpus Catalogue =

{| class="wikitable"
|-
! Directory !! Description !! System !! Install Date !! Maintainer
|-
| conll17 || [http://hdl.handle.net/11234/1-1989 Corpora for 46 Languages (from Wikipedia and Common Crawl)] || Abel, Taito || November 2017 || Jenna Kaverna
|-
| EngC3 || [http://urn.nb.no/URN:NBN:no-60569 130 Billion Tokens of ‘Clean’ Text from the Common Crawl] || Abel, Taito || April 2018 || Stephan Oepen
|}

= Many Languages: The Collection of Open Parallel Corpora (Helsinki) =

[[Corpora/OPUS|OPUS]] provides parallel texts in many languages extracted from
a broad range of freely available sources
In terms of the NLPL-internal
project structure, it is actually a task of its own (because there are
additional on-line services connected to OPUS), hence there is a
[[Corpora/OPUS|separate wiki page]] with additional information on OPUS.

= Many Languages: The CoNLL 2017 Text Collection (Turku) =

The collection consists of 90B words in 45 languages gathered from CommonCrawl and Wikipedia dumps. The data ranges from around 9B words for English to 28K words of Old Church Slavonic. A number of filtering and deduplication steps have been applied, to arrive at a relatively clean extracted text. The texts are segmented and fully parsed with the UDPipe v1.1 parser. The processing pipeline is described in greater detail in Section 2.2 of this paper: [http://universaldependencies.org/conll17/proceedings/pdf/K17-3001.pdf] This collection was used as supporting data in the 2017 and 2018 CoNLL shared tasks. In addition some of the word2vec and ELMo embeddings distributed within NLPL are based on this data.

The raw, unprocessed version of the crawls will be available (at least) on Taito and re-parsing this data with one of the top-ranking parsers in the 2018 Shared Task is under consideration.

= English: 130 Billion Words Extracted from the Common Crawl (Oslo) =

= English: Two Variants of Text Extraction from Wikipedia (Oslo) =

Corpora/home

2019-02-04T16:04:29Z

Figint: /* Many Languages: The CoNLL 2017 Text Collection (Turku) */

= Background =

NLPL creates and makes available various very large collections of textual data, for example drawing on Wikipedia and the Common Crawl.

These data resources are available from the connected infrastructure, below the project directory
<tt>/projects/nlpl/data/corpora/</tt> or <tt>/proj/nlpl/data/corpora/</tt> on Abel and Taito, respectively.
In early 2018, the NLPL corpora resources are not yet replicated across the two systems; please see
the summary table below to determine which resource is available on which system.

= Corpus Catalogue =

{| class="wikitable"
|-
! Directory !! Description !! System !! Install Date !! Maintainer
|-
| conll17 || [http://hdl.handle.net/11234/1-1989 Corpora for 46 Languages (from Wikipedia and Common Crawl)] || Abel, Taito || November 2017 || Jenna Kaverna
|-
| EngC3 || [http://urn.nb.no/URN:NBN:no-60569 130 Billion Tokens of ‘Clean’ Text from the Common Crawl] || Abel, Taito || April 2018 || Stephan Oepen
|}

= Many Languages: The Collection of Open Parallel Corpora (Helsinki) =

[[Corpora/OPUS|OPUS]] provides parallel texts in many languages extracted from
a broad range of freely available sources
In terms of the NLPL-internal
project structure, it is actually a task of its own (because there are
additional on-line services connected to OPUS), hence there is a
[[Corpora/OPUS|separate wiki page]] with additional information on OPUS.

= Many Languages: The CoNLL 2017 Text Collection (Turku) =

The collection consists of 90B words in 45 languages gathered from CommonCrawl and Wikipedia dumps. The data ranges from around 9B words for English to 28K words of Old Church Slavonic. A number of filtering and deduplication steps have been applied, to arrive at a relatively clean extracted text. The texts are segmented and fully parsed with the UDPipe v1.1 parser. The processing pipeline is described in greater detail in Section 2.2 of this paper: [http://universaldependencies.org/conll17/proceedings/pdf/K17-3001.pdf] This collection was used as supporting data in the 2017 and 2018 CoNLL shared tasks. In addition some of the word2vec and ELMo embeddings distributed within NLPL are based on this data.

= English: 130 Billion Words Extracted from the Common Crawl (Oslo) =

= English: Two Variants of Text Extraction from Wikipedia (Oslo) =

Corpora/home

2019-02-04T15:59:11Z

Figint: /* Many Languages: The CoNLL 2017 Text Collection (Turku) */

= Background =

NLPL creates and makes available various very large collections of textual data, for example drawing on Wikipedia and the Common Crawl.

These data resources are available from the connected infrastructure, below the project directory
<tt>/projects/nlpl/data/corpora/</tt> or <tt>/proj/nlpl/data/corpora/</tt> on Abel and Taito, respectively.
In early 2018, the NLPL corpora resources are not yet replicated across the two systems; please see
the summary table below to determine which resource is available on which system.

= Corpus Catalogue =

{| class="wikitable"
|-
! Directory !! Description !! System !! Install Date !! Maintainer
|-
| conll17 || [http://hdl.handle.net/11234/1-1989 Corpora for 46 Languages (from Wikipedia and Common Crawl)] || Abel, Taito || November 2017 || Jenna Kaverna
|-
| EngC3 || [http://urn.nb.no/URN:NBN:no-60569 130 Billion Tokens of ‘Clean’ Text from the Common Crawl] || Abel, Taito || April 2018 || Stephan Oepen
|}

= Many Languages: The Collection of Open Parallel Corpora (Helsinki) =

[[Corpora/OPUS|OPUS]] provides parallel texts in many languages extracted from
a broad range of freely available sources
In terms of the NLPL-internal
project structure, it is actually a task of its own (because there are
additional on-line services connected to OPUS), hence there is a
[[Corpora/OPUS|separate wiki page]] with additional information on OPUS.

= Many Languages: The CoNLL 2017 Text Collection (Turku) =

The collection consists of 90B words in 45 languages gathered from CommonCrawl and Wikipedia dumps. The data ranges from around 9B words for English to 28K words of Old Church Slavonic. A number of filtering and deduplication steps have been applied, to arrive at a relatively clean extracted text. These are described in greater detail in Section 2.2 of this paper: [http://universaldependencies.org/conll17/proceedings/pdf/K17-3001.pdf]

= English: 130 Billion Words Extracted from the Common Crawl (Oslo) =

= English: Two Variants of Text Extraction from Wikipedia (Oslo) =

Corpora/home

2019-02-04T15:56:25Z

Figint: /* Many Languages: The CoNLL 2017 Text Collection (Turku) */

= Background =

NLPL creates and makes available various very large collections of textual data, for example drawing on Wikipedia and the Common Crawl.

These data resources are available from the connected infrastructure, below the project directory
<tt>/projects/nlpl/data/corpora/</tt> or <tt>/proj/nlpl/data/corpora/</tt> on Abel and Taito, respectively.
In early 2018, the NLPL corpora resources are not yet replicated across the two systems; please see
the summary table below to determine which resource is available on which system.

= Corpus Catalogue =

{| class="wikitable"
|-
! Directory !! Description !! System !! Install Date !! Maintainer
|-
| conll17 || [http://hdl.handle.net/11234/1-1989 Corpora for 46 Languages (from Wikipedia and Common Crawl)] || Abel, Taito || November 2017 || Jenna Kaverna
|-
| EngC3 || [http://urn.nb.no/URN:NBN:no-60569 130 Billion Tokens of ‘Clean’ Text from the Common Crawl] || Abel, Taito || April 2018 || Stephan Oepen
|}

= Many Languages: The Collection of Open Parallel Corpora (Helsinki) =

[[Corpora/OPUS|OPUS]] provides parallel texts in many languages extracted from
a broad range of freely available sources
In terms of the NLPL-internal
project structure, it is actually a task of its own (because there are
additional on-line services connected to OPUS), hence there is a
[[Corpora/OPUS|separate wiki page]] with additional information on OPUS.

= Many Languages: The CoNLL 2017 Text Collection (Turku) =

The collection consists of 90B words in 45 languages gathered from CommonCrawl and Wikipedia dumps. The data ranges from around 9B words for English to 28K words of Old Church Slavonic.

= English: 130 Billion Words Extracted from the Common Crawl (Oslo) =

= English: Two Variants of Text Extraction from Wikipedia (Oslo) =

Corpora/home

2019-02-04T15:55:02Z

Figint: /* Many Languages: The CoNLL 2017 Text Collection (Turku) */

Corpora/home

2019-02-04T15:53:39Z

Figint: /* Many Languages: The CoNLL 2017 Text Collection (Turku) */