Vectors/home - Revision history

Oe: /* Background */

2021-01-12T22:23:50Z

‎Background

Oe: /* Background */

2021-01-12T22:23:09Z

‎Background

Andreku: /* Background */

2021-01-12T20:03:44Z

‎Background

Oe: /* Background */

2021-01-12T18:43:40Z

‎Background

Oe at 22:31, 2 January 2020

2020-01-02T22:31:43Z

Andreku: Version 2.0 and moving to Saga

2019-12-29T18:41:03Z

Version 2.0 and moving to Saga

Oe: /* Repository Contents */

2019-12-28T21:38:00Z

‎Repository Contents

Andreku: /* Repository Contents */

2019-12-22T23:54:53Z

‎Repository Contents

Andreku: /* Repository Contents */

2019-12-22T23:54:43Z

‎Repository Contents

Oe: /* Future Work */

2019-12-20T16:13:24Z

‎Future Work

@@ Line 18: / Line 18: @@
 version 1.0.
 In March 2018, version 1.1 supersedes this initial release, adding a large number
-of models and languages (including those from the
+of models and languages (including those from the 2017
-[http://hdl.handle.net/11234/1-1989 UD parsing task])
+[http://hdl.handle.net/11234/1-1989 UD parsing shared task])
 and re-packaging the models from the original release in a more standardized
 format (see below).

@@ Line 22: / Line 22: @@
 and re-packaging the models from the original release in a more standardized
 format (see below).
-In December 2019, version 2.0 was released, which added BERT and ELMo models, made metadata more consistent
+In December 2019, version 2.0 was released, which added BERT and ELMo models (including large monolingual
 and ensured that binary format models are always provided
 (to increase the loading speed, in comparison to the models stored as plain text).
 = Repository Contents =

@@ Line 25: / Line 25: @@
 and ensured that binary format models are always provided
 (to increase the loading speed, in comparison to the models stored as plain text).
 = Repository Contents =

@@ Line 1: / Line 1: @@
 = Background =
-The purpose of the NLPL repository of word vectors/embeddings (which can comprise both ‘classic’, count-based and ‘modern’, dense models, including the contextualized ones) is
+The purpose of the NLPL repository of word vectors (or embeddings, which can comprise both ‘classic’, count-based and ‘modern’, dense models, including the contextualized ones) is
 to make available a large and carefully curated collection of large-scale distributional semantic models for many languages.
 For general background, please see

@@ Line 8: / Line 8: @@
 For interactive exploration and download of the repository, there is an
 [http://vectors.nlpl.eu/repository/ on-line explorer].
-The underlying data is stored in the NLPL project directory below
+The underlying data is stored in the NLPL community directory below
 <tt>/cluster/shared/nlpl/data/vectors/</tt> (on Saga)
-and <tt>/proj/nlpl/data/vectors/</tt> (on Taito).
+and <tt>/projappl/nlpl/data/vectors/</tt> (on Puhti).
 The repository is versioned, in the sense of assigning release numbers to different
 stages of repository construction.
@@ Line 37: / Line 37: @@
 (data sources), <tt>algorithms</tt> (model creation tools), and
 <tt>models</tt> (resulting sets of word vectors).
-NLPL users with access to Abel and Taito can read the
+NLPL users with access to Saga and Puhti can read the
 catalogue file directly from the
-project directory, for example when executing a series of experiments
+NLPL community directory, for example when executing a series of experiments
 that make use of different pre-trained sets of word vectors.
 Further
@@ Line 59: / Line 59: @@
 To avoid data duplication, it is recommended to load models from the NLPL repository
-directly from the NLPL project directory, when working on Saga or Taito.
+directly from the NLPL community directory, when working on Saga or Puhti.
 Repository entries are uniformly packaged as <tt>.zip</tt> compressed archives, but
 the uniform naming scheming makes it possible to directly read one or more of the

@@ Line 1: / Line 1: @@
 = Background =
-The purpose of the NLPL repository of word vectors/embeddings (which can comprise both ‘classic’, count-based and ‘modern’, dense models) is
+The purpose of the NLPL repository of word vectors/embeddings (which can comprise both ‘classic’, count-based and ‘modern’, dense models, including the contextualized ones) is
 to make available a large and carefully curated collection of large-scale distributional semantic models for many languages.
 For general background, please see
@@ Line 9: / Line 9: @@
 [http://vectors.nlpl.eu/repository/ on-line explorer].
 The underlying data is stored in the NLPL project directory below
-<tt>/projects/nlpl/data/vectors/</tt> (on Abel)
+<tt>/cluster/shared/nlpl/data/vectors/</tt> (on Saga)
 and <tt>/proj/nlpl/data/vectors/</tt> (on Taito).
 The repository is versioned, in the sense of assigning release numbers to different
@@ Line 22: / Line 22: @@
 and re-packaging the models from the original release in a more standardized
 format (see below).
 = Repository Contents =
@@ Line 28: / Line 31: @@
 for programmatic access in the repository ''catalogue'', which is represented
 as a JSON file in the top-level repository directory, with catalogue names
-corresponding to each repository version, e.g. <tt>/projects/nlpl/data/vectors/10.json</tt>
+corresponding to each repository version, e.g. <tt>/cluster/shared/nlpl/data/vectors/20.json</tt>
-(on Abel) for the initial repository release.
+(on Saga) for the current repository release.
 The catalogue contains three top-level sections, one each for <tt>corpora</tt>
@@ Line 50: / Line 53: @@
 the specific contents; a <tt>README</tt> file included
 with each model entry provides a life-time unique identifier, e.g.
-<tt>http://vectors.nlpl.eu/repository/11/3.zip</tt> for
+<tt>http://vectors.nlpl.eu/repository/20/3.zip</tt> for
-model #3 in the 1.1 release of the repository.
+model #3 in the 2.0 release of the repository.
 = Using NLPL Models In-Situ =
 To avoid data duplication, it is recommended to load models from the NLPL repository
-directly from the NLPL project directory, when working on Abel or Taito.
+directly from the NLPL project directory, when working on Saga or Taito.
 Repository entries are uniformly packaged as <tt>.zip</tt> compressed archives, but
 the uniform naming scheming makes it possible to directly read one or more of the
@@ Line 66: / Line 69: @@
   import zipfile
   import gensim
-  repository = "/projects/nlpl/data/vectors/11"
+  repository = "/cluster/shared/nlpl/data/vectors/20"
   with zipfile.ZipFile(repository + "/30.zip", "r") as archive:
     stream = archive.open("model.txt")
@@ Line 81: / Line 84: @@
 = Future Work =
-# Prepare Version 2.0, now with <code>/cluster/shared/nlpl/data/vectors/</code> as the master copy.
+# Prepare Version 2.0, now with <code>/cluster/shared/nlpl/data/vectors/</code> as the master copy (DONE).
-# The life-time <tt>handle</tt> for each model should be included in the JSON catalogue (in addition to being listed in the <tt>README</tt> file).
+# The life-time <tt>handle</tt> for each model should be included in the JSON catalogue (in addition to being listed in the <tt>README</tt> file) (DONE).
-# For classic models, redundantly add binary binary <code>model.bin</code> for faster loading.
+# For classic models, redundantly add binary binary <code>model.bin</code> for faster loading (DONE).
 # Each corpus should be listed as a separate entry; corpus combinations go into the array-valued <tt>corpus</tt> property on models (DONE).
-# Where applicable, there should be an array-valued <tt>documentation</tt> field (of string, typically URLs) on corpora and models.
+# Where applicable, there should be an array-valued <tt>documentation</tt> field (of string, typically URLs) on corpora and models (DONE).
-# The <tt>maintainers</tt> property may be over-promising, as often third-party models are in practice unmaintained; maybe rename to <tt>creator</tt>?
+# The <tt>maintainers</tt> property may be over-promising, as often third-party models are in practice unmaintained; maybe rename to <tt>creator</tt> (DONE).
 # Document (and possibly re-design) the metadata scheme; maybe invent a fourth category for a <tt>process</tt> applied prior model training.

@@ Line 38: / Line 38: @@
 project directory, for example when executing a series of experiments
 that make use of different pre-trained sets of word vectors.
+Further
-[[Repository catalogue fields]]
+[http://wiki.nlpl.eu/index.php/Vectors/metadata documentation] of the catalogue metadata is available
 Each repository entry (i.e. set of word vectors, or ‘model’) is