Difference between revisions of "Eosc/easybuild/modules"

From Nordic Language Processing Laboratory
Jump to: navigation, search
(List of modules)
(Others)
(8 intermediate revisions by the same user not shown)
Line 26: Line 26:
 
Those with "foss" instead of "gomkl" are CPU-agnostic and will run on machines with AMD CPUs (for example, on ''Fox'').
 
Those with "foss" instead of "gomkl" are CPU-agnostic and will run on machines with AMD CPUs (for example, on ''Fox'').
  
Further on, we just use the placeholder '''ARCH''', replace it with either "gomkl" or "foss", depending on which machine you are working on.
+
The next element in the module name after "foss" or "gomkl" is the virtual laboratory (stack) version: for example, "2019b" or "2021a".
 +
Modules from different stack versions are incompatible  with each other: you cannot load a module from "foss-2019b" and a module from "foss-2021a" simultaneously.
 +
Currently, "2021a" version is the recommended one, with the most up-to-date software built with the most up-to-date compilers.
 +
 
 +
Further on, we just use the placeholder '''ARCH''', replace it with "gomkl" or "foss" and "2019b" or "2021a", depending on which machine you are working on and what stack version you want to use.
 +
Some modules also have the Python version specified in their names (for example, "''nlpl-numpy-1.23.1-foss-2021a-Python-3.9.5''").
 +
For stack version "2019b" it is usually Python 3.7.4, for stack version "2021a" it is usually Python 3.9.5.
 +
Check the output of the '''module avail''' command for the exact module names.
  
 
=== "Bundle" modules ===
 
=== "Bundle" modules ===
 
These are the modules with the most cryptic names. Each of them contains a bunch of software pieces (Python packages, as a rule).
 
These are the modules with the most cryptic names. Each of them contains a bunch of software pieces (Python packages, as a rule).
 
Many of these modules will be loaded automatically as dependencies of the regular modules, but sometimes they can be useful themselves.
 
Many of these modules will be loaded automatically as dependencies of the regular modules, but sometimes they can be useful themselves.
 +
They have their own bundle versions: "2021.01", "2022.01", etc (further specified as '''VERS''').
 
Here are the details:
 
Here are the details:
  
* '''nlpl-python-candy/2021.01-ARCH-2019b-Python-3.7.4''': various utility packages not directly related to NLP
+
* '''nlpl-python-candy/VERS-ARCH''': various utility packages not directly related to NLP
** tqdm 4.62.3
+
** tqdm
** pydot 1.4.2
+
** pydot
** smart_open 5.2.1
+
** smart_open
** cached-property 1.5.2
+
** cached-property
** filelock 3.0.12
+
** filelock
** regex 2021.10.23
+
** regex
** sacremoses 0.0.46
+
** sacremoses
** mpi4py 3.1.1
+
** mpi4py
** jsonlines 2.0.0
+
** jsonlines
** typing_extensions 3.7.4.3
+
** typing_extensions
* '''nlpl-nlptools/2021.01-ARCH-2019b-Python-3.7.4''': various utility packages related to NLP
+
** packaging
** conllu 4.4.1
+
* '''nlpl-nlptools/VERS-ARCH''': various utility packages related to NLP
** seqeval 1.2.2
+
** conllu
** langdetect 1.0.9
+
** seqeval
** leven 1.0.4
+
** langdetect
* '''nlpl-scipy-ecosystem/2021.01-ARCH-2019b-Python-3.7.4''': everything that constitutes the [https://scipy.org/ SciPy ecosystem]. Too many packages to enumerate them all, but the most important are:
+
** leven
** scipy 1.5.4
+
* '''nlpl-scipy-ecosystem/VERS-ARCH''': everything that constitutes the [https://scipy.org/ SciPy ecosystem]. Too many packages to enumerate them all, but the most important are:
** pandas 1.2.1
+
** scipy
** matplotlib 3.1.2
+
** pandas
** ipython 7.11.1
+
** matplotlib
** jupyter_core 4.6.1
+
** ipython
** jupyter_client 5.3.4
+
** jupyter_core
** networkx 2.5.1
+
** jupyter_client
** sympy 1.7.1
+
** networkx
 +
** sympy
  
 
=== "Regular" modules ===
 
=== "Regular" modules ===
 
These are more obvious modules, each one gives you one software piece:
 
These are more obvious modules, each one gives you one software piece:
  
* '''nlpl-cython/0.29.21-ARCH-2019b-Python-3.7.4''': [http://cython.org/ Cython] 0.29.21
+
==== Most important ====
* '''nlpl-datasets/1.17-ARCH-2019b-Python-3.7.4''': [https://github.com/huggingface/datasets HuggingFace Datasets] 1.17
+
* '''nlpl-pytorch/1.6.0-ARCH-cuda-10.1.243-Python-3.7.4''': [https://pytorch.org/ PyTorch] 1.6.0 (for CUDA 10)
 +
* '''nlpl-pytorch/1.7.1-ARCH-cuda-10.1.243-Python-3.7.4''': [https://pytorch.org/ PyTorch] 1.7.1 (for CUDA 10)
 +
* '''nlpl-pytorch/1.7.1-ARCH-cuda-11.1.1-Python-3.7.4''': [https://pytorch.org/ PyTorch] 1.7.1 (for CUDA 11)
 +
* '''nlpl-pytorch/1.11.0-ARCH-cuda-11.3.1-Python-3.9.5''': [https://pytorch.org/ PyTorch] 1.11.0 (for CUDA 11)
 +
 
 +
* '''nlpl-tensorflow/1.15.2-ARCH-cuda-10.1.243-Python-3.7.4''': [https://www.tensorflow.org/ TensorFlow] 1.15.2 (for CUDA 10)
 +
* '''nlpl-tensorflow/2.3.2-ARCH-cuda-10.1.243-Python-3.7.4''': [https://www.tensorflow.org/ TensorFlow] 2.3.2 (for CUDA 10)
 +
* '''nlpl-tensorflow/2.6.2-ARCH-cuda-11.1.1-Python-3.7.4''': [https://www.tensorflow.org/ TensorFlow] 2.6.2 (for CUDA 11)
 +
* '''nlpl-tensorflow/2.6.5-ARCH-cuda-11.3.1-Python-3.9.5''': [https://www.tensorflow.org/ TensorFlow] 2.6.5 (for CUDA 11)
 +
 
 +
==== Others ====
 +
* '''nlpl-cython/VERS-ARCH''': [http://cython.org/ Cython]
 +
* '''nlpl-datasets/VERS-ARCH''': [https://github.com/huggingface/datasets HuggingFace Datasets]
 +
* '''nlpl-gensim/VERS-ARCH''': [https://github.com/RaRe-Technologies/gensim Gensim]
 +
* '''nlpl-horovod/VERS-ARCH''': [https://github.com/horovod/horovod Horovod]
 +
* '''nlpl-huggingface-hub/VERS-ARCH-2019b-Python-3.7.4''': [https://pypi.org/project/huggingface-hub/ HuggingFace Hub]
 +
* '''nlpl-nltk/VERS-ARCH''': [https://www.nltk.org/ NLTK], together with '''all''' the corpora and datasets (no need to download them separately!)
 +
* '''nlpl-numpy/VERS-ARCH''': [https://numpy.org/ NumPy]
 +
* '''nlpl-pytorch-lightning/VERS-ARCH-cuda-11.3.1''': [https://www.pytorchlightning.ai/ PyTorch Lightning]
 +
* '''nlpl-scikit-bundle/VERS-ARCH''': [https://scikit-learn.org/ Scikit-Learn]
 +
* '''nlpl-sentencepiece/VERS-ARCH''': [https://github.com/google/sentencepiece SentencePiece]
 +
* '''nlpl-simple_elmo/VERS-ARCH''': [https://pypi.org/project/simple-elmo/ Simple_elmo]
 +
* '''nlpl-stanza/VERS-ARCH''': [https://stanfordnlp.github.io/stanza/ Stanza]
 +
* '''nlpl-tensorboard/VERS-ARCH''': [https://github.com/tensorflow/tensorboard TensorBoard]
 +
* '''nlpl-tokenizers/VERS-ARCH''': [https://github.com/huggingface/tokenizers HuggingFace Tokenizers]
 +
* '''nlpl-torchmetrics/VERS-ARCH''': [https://pypi.org/project/torchmetrics/ TorchMetrics]
 +
* '''nlpl-torchtext/VERS-ARCH''': [https://pypi.org/project/torchtext/ TorchText]
 +
* '''nlpl-transformers/VERS-ARCH''': [https://huggingface.co/transformers/ HuggingFace Transformers]
 +
* '''nlpl-wandb/VERS-ARCH''': [https://pypi.org/project/wandb/ Weights and Biases (wandb)]
 +
 
 
* '''nlpl-dllogger/0.1.0-ARCH-2019b-Python-3.7.4''': [https://github.com/NVIDIA/dllogger DLLogger] 0.1.0 ''(status unclear)''
 
* '''nlpl-dllogger/0.1.0-ARCH-2019b-Python-3.7.4''': [https://github.com/NVIDIA/dllogger DLLogger] 0.1.0 ''(status unclear)''
* '''nlpl-gensim/3.8.3-ARCH-2019b-Python-3.7.4''': [https://github.com/RaRe-Technologies/gensim Gensim] 3.8.3
 
* '''nlpl-horovod/0.20.3-ARCH-2019b-tensorflow-1.15.2-Python-3.7.4''': [https://github.com/horovod/horovod Horovod] 0.20.3
 
* '''nlpl-horovod/0.23.0-ARCH-2019b-tensorflow-2.6.2-Python-3.7.4''': [https://github.com/horovod/horovod Horovod] 0.23.0
 
* '''nlpl-huggingface-hub/0.4.0-ARCH-2019b-Python-3.7.4''': [https://pypi.org/project/huggingface-hub/ HuggingFace Hub] 0.4.0
 
* '''nlpl-nltk/3.5-ARCH-2019b-Python-3.7.4''': [https://www.nltk.org/ NLTK] 3.5, together with '''all''' the corpora and datasets (no need to download them separately!)
 
* '''nlpl-numpy/1.18.1-ARCH-2019b-Python-3.7.4''': [https://numpy.org/ NumPy] 1.18.1
 
 
* '''nlpl-nvidia-bert/20.06.8-ARCH-2019b-tensorflow-1.15.2-Python-3.7.4''': [https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT NVIDIA's BERT implementation] for TensorFlow 1 ''(status unclear)''
 
* '''nlpl-nvidia-bert/20.06.8-ARCH-2019b-tensorflow-1.15.2-Python-3.7.4''': [https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT NVIDIA's BERT implementation] for TensorFlow 1 ''(status unclear)''
* '''nlpl-pytorch/1.6.0-ARCH-2019b-cuda-10.1.243-Python-3.7.4''': [https://pytorch.org/ PyTorch] 1.6.0
 
* '''nlpl-pytorch/1.7.1-ARCH-2019b-cuda-10.1.243-Python-3.7.4''': [https://pytorch.org/ PyTorch] 1.7.1
 
* '''nlpl-pytorch/1.7.1-ARCH-2019b-cuda-11.1.1-Python-3.7.4''': [https://pytorch.org/ PyTorch] 1.7.1 (for CUDA 11)
 
* '''nlpl-scikit-bundle/0.22.2.post1-ARCH-2019b-Python-3.7.4''': [https://scikit-learn.org/ Scikit-Learn] 0.22.2
 
* '''nlpl-simple_elmo/0.9.0-ARCH-2019b-Python-3.7.4''': [https://pypi.org/project/simple-elmo/ Simple_elmo] 0.9.0
 
* '''nlpl-stanza/1.1.1-ARCH-2019b-Python-3.7.4''': [https://stanfordnlp.github.io/stanza/ Stanza] 1.1.1
 
* '''nlpl-stanza/1.3.0-ARCH-2019b-Python-3.7.4''': [https://stanfordnlp.github.io/stanza/ Stanza] 1.3.0
 
* '''nlpl-tensorflow/1.15.2-ARCH-2019b-cuda-10.1.243-Python-3.7.4''': [https://www.tensorflow.org/ TensorFlow] 1.15.2
 
* '''nlpl-tensorflow/2.3.2-ARCH-2019b-cuda-10.1.243-Python-3.7.4''': [https://www.tensorflow.org/ TensorFlow] 2.3.2
 
* '''nlpl-tensorflow/2.6.2-ARCH-2019b-cuda-11.1.1-Python-3.7.4''': [https://www.tensorflow.org/ TensorFlow] 2.6.2 (for CUDA 11)
 
* '''nlpl-tokenizers/0.10.2-ARCH-2019b-Python-3.7.4''': [https://github.com/huggingface/tokenizers HuggingFace Tokenizers] 0.10.2
 
* '''nlpl-transformers/4.5.1-ARCH-2019b-Python-3.7.4''': [https://huggingface.co/transformers/ HuggingFace Transformers] 4.5.1
 
* '''nlpl-transformers/4.14.1-ARCH-2019b-Python-3.7.4''': [https://huggingface.co/transformers/ HuggingFace Transformers] 4.14.1
 
* '''nlpl-wandb/0.12.6-ARCH-2019b-Python-3.7.4''': [https://pypi.org/project/wandb/ Weights and Biases (wandb)] 0.12.6
 
* '''sentencepiece/0.1.94-ARCH-2019b-Python-3.7.4''': [https://github.com/google/sentencepiece SentencePiece] 0.1.94
 
* '''sentencepiece/0.1.96-ARCH-2019b-Python-3.7.4''': [https://github.com/google/sentencepiece SentencePiece] 0.1.96
 
  
 
= Source =
 
= Source =
  
 
Currently, the virtual laboratory is generated using EasyBuild, all the code and easyconfigs available [https://source.coderefinery.org/nlpl/easybuild here].
 
Currently, the virtual laboratory is generated using EasyBuild, all the code and easyconfigs available [https://source.coderefinery.org/nlpl/easybuild here].

Revision as of 22:25, 23 August 2022

NLPL virtual laboratory

The laboratory is a reproducible custom-built set of NLP software. It is currently installed on Saga, Fox, and Puhti HPC clusters.

- To use on Saga: run the following command (can be put in the ~/.bashrc file to be run automatically at login):

module use -a /cluster/shared/nlpl/software/eb/etc/all/

- To use on Fox: run the following command (can be put in the ~/.bashrc file to be run automatically at login):

module use -a /fp/projects01/ec30/software/easybuild/modules/all/

After that, the "nlpl"-branded modules will be available via module avail, module load, etc.

It is highly recommended to use them, instead of installing a copy in one's own home directory.

List of modules

From time to time, updated modules with newer software versions will be added, but the older modules will never be removed (for reproducibility).

Note that the modules which have "gomkl" in their names are built using Intel Math Kernel Library, making them significantly faster in CPU tasks with Intel processors (for example, on Saga).

Those with "foss" instead of "gomkl" are CPU-agnostic and will run on machines with AMD CPUs (for example, on Fox).

The next element in the module name after "foss" or "gomkl" is the virtual laboratory (stack) version: for example, "2019b" or "2021a". Modules from different stack versions are incompatible with each other: you cannot load a module from "foss-2019b" and a module from "foss-2021a" simultaneously. Currently, "2021a" version is the recommended one, with the most up-to-date software built with the most up-to-date compilers.

Further on, we just use the placeholder ARCH, replace it with "gomkl" or "foss" and "2019b" or "2021a", depending on which machine you are working on and what stack version you want to use. Some modules also have the Python version specified in their names (for example, "nlpl-numpy-1.23.1-foss-2021a-Python-3.9.5"). For stack version "2019b" it is usually Python 3.7.4, for stack version "2021a" it is usually Python 3.9.5. Check the output of the module avail command for the exact module names.

"Bundle" modules

These are the modules with the most cryptic names. Each of them contains a bunch of software pieces (Python packages, as a rule). Many of these modules will be loaded automatically as dependencies of the regular modules, but sometimes they can be useful themselves. They have their own bundle versions: "2021.01", "2022.01", etc (further specified as VERS). Here are the details:

  • nlpl-python-candy/VERS-ARCH: various utility packages not directly related to NLP
    • tqdm
    • pydot
    • smart_open
    • cached-property
    • filelock
    • regex
    • sacremoses
    • mpi4py
    • jsonlines
    • typing_extensions
    • packaging
  • nlpl-nlptools/VERS-ARCH: various utility packages related to NLP
    • conllu
    • seqeval
    • langdetect
    • leven
  • nlpl-scipy-ecosystem/VERS-ARCH: everything that constitutes the SciPy ecosystem. Too many packages to enumerate them all, but the most important are:
    • scipy
    • pandas
    • matplotlib
    • ipython
    • jupyter_core
    • jupyter_client
    • networkx
    • sympy

"Regular" modules

These are more obvious modules, each one gives you one software piece:

Most important

  • nlpl-pytorch/1.6.0-ARCH-cuda-10.1.243-Python-3.7.4: PyTorch 1.6.0 (for CUDA 10)
  • nlpl-pytorch/1.7.1-ARCH-cuda-10.1.243-Python-3.7.4: PyTorch 1.7.1 (for CUDA 10)
  • nlpl-pytorch/1.7.1-ARCH-cuda-11.1.1-Python-3.7.4: PyTorch 1.7.1 (for CUDA 11)
  • nlpl-pytorch/1.11.0-ARCH-cuda-11.3.1-Python-3.9.5: PyTorch 1.11.0 (for CUDA 11)
  • nlpl-tensorflow/1.15.2-ARCH-cuda-10.1.243-Python-3.7.4: TensorFlow 1.15.2 (for CUDA 10)
  • nlpl-tensorflow/2.3.2-ARCH-cuda-10.1.243-Python-3.7.4: TensorFlow 2.3.2 (for CUDA 10)
  • nlpl-tensorflow/2.6.2-ARCH-cuda-11.1.1-Python-3.7.4: TensorFlow 2.6.2 (for CUDA 11)
  • nlpl-tensorflow/2.6.5-ARCH-cuda-11.3.1-Python-3.9.5: TensorFlow 2.6.5 (for CUDA 11)

Others

  • nlpl-dllogger/0.1.0-ARCH-2019b-Python-3.7.4: DLLogger 0.1.0 (status unclear)
  • nlpl-nvidia-bert/20.06.8-ARCH-2019b-tensorflow-1.15.2-Python-3.7.4: NVIDIA's BERT implementation for TensorFlow 1 (status unclear)

Source

Currently, the virtual laboratory is generated using EasyBuild, all the code and easyconfigs available here.