From Nordic Language Processing Laboratory
Revision as of 13:27, 5 November 2021 by Andreku (talk | contribs) (List of modules)
Jump to: navigation, search

NLPL virtual laboratory

The laboratory is a reproducible custom-built set of NLP software. It is currently installed on Saga and Puhti HPC clusters.

To use on Saga: run the following command on (can be put in the ~/.bashrc file to be run automatically at login):

module use -a /cluster/shared/nlpl/software/eb/etc/all/

After that, the "nlpl"-branded modules will be available via module avail, module load, etc.

List of modules

Note that the modules which have "gomkl" in their names are built using Intel Math Kernel Library, making them significantly faster in CPU tasks with Intel processors.

"Bundle" modules

These are the modules with the most cryptic names. Each of them contains a bunch of software pieces (Python packages, as a rule). Here are the details:

  • nlpl-python-candy/2021.01-gomkl-2019b-Python-3.7.4
  • nlpl-nlptools/2021.01-gomkl-2019b-Python-3.7.4
  • nlpl-scikit-bundle/0.22.2.post1-gomkl-2019b-Python-3.7.4
  • nlpl-scipy-ecosystem/2021.01-gomkl-2019b-Python-3.7.4

"Regular" modules

These are more obvious modules, each one gives you one software piece:

  • nlpl-cython/0.29.21-gomkl-2019b-Python-3.7.4
  • nlpl-dllogger/0.1.0-gomkl-2019b-Python-3.7.4
  • nlpl-gensim/3.8.3-gomkl-2019b-Python-3.7.4
  • nlpl-horovod/0.20.3-gomkl-2019b-tensorflow-1.15.2-Python-3.7.4
  • nlpl-nltk/3.5-gomkl-2019b-Python-3.7.4
  • nlpl-numpy/1.18.1-gomkl-2019b-Python-3.7.4
  • nlpl-nvidia-bert/20.06.8-gomkl-2019b-tensorflow-1.15.2-Python-3.7.4
  • nlpl-pytorch/1.6.0-gomkl-2019b-cuda-10.1.243-Python-3.7.4
  • nlpl-simple_elmo/0.6.0-gomkl-2019b-Python-3.7.4
  • nlpl-stanza/1.1.1-gomkl-2019b-Python-3.7.4
  • nlpl-tensorflow/1.15.2-gomkl-2019b-cuda-10.1.243-Python-3.7.4
  • nlpl-tensorflow/2.3.2-gomkl-2019b-cuda-10.1.243-Python-3.7.4
  • nlpl-tokenizers/0.10.2-gomkl-2019b-Python-3.7.4
  • nlpl-transformers/4.5.1-gomkl-2019b-Python-3.7.4
  • nlpl-wandb/0.12.6-gomkl-2019b-Python-3.7.4
  • sentencepiece/0.1.94-gomkl-2019b-Python-3.7.4


Currently, the virtual laboratory is generated using EasyBuild, all the code and easyconfigs available here.