Difference between revisions of "Eosc/easybuild"
(→Important stuff to remember) |
(→Background) |
||
| Line 5: | Line 5: | ||
In principle, | In principle, | ||
As part of the NLPL use case in EOSC-Nordic, we are evaluating EasyBuild for this purpose. | As part of the NLPL use case in EOSC-Nordic, we are evaluating EasyBuild for this purpose. | ||
| + | |||
| + | = Desk Pilot = | ||
| + | |||
| + | To get started, we set out to re-create one common stack of NLPL | ||
| + | modules in a fully automated EasyBuild configuration, viz. | ||
| + | Python 3.7.4, NumPy 1.18.1, the SciPy Bundle (SciPy 1.4.1, SciKit-Learn 0.22.1, iPython 7.11.1, MatPlotLib 3.1.2, Pandas 0.23.1), and TensorFlow 1.15.2). | ||
| + | For additional thrill, there should be two versions of NumPy, one installed | ||
| + | with the MKL backend, the other without (using the default, which we | ||
| + | believe is OpenBLAS). | ||
| + | Ideally, this choice near the bottom of the dependency tree | ||
| + | should not propagate into the higher-level modules, i.e. we would | ||
| + | hope to have only once instance of the SciPy bundle or TensorFlow, | ||
| + | and they would interoperate seamlessly with either choice for NumPy. | ||
= Important stuff to remember = | = Important stuff to remember = | ||
Revision as of 15:14, 20 October 2020
Background
The goal is to organize provisioning of software (for NLP research) in a manner that makes it possible and cost-efficient to maintain the exact same software stack on multiple systems. Here, systems initially means different superclusters, e.g. Puhti in Finland and Saga in Norway; sometime in 2021, we anticipate to additionally support the LUMI environment. In principle, As part of the NLPL use case in EOSC-Nordic, we are evaluating EasyBuild for this purpose.
Desk Pilot
To get started, we set out to re-create one common stack of NLPL modules in a fully automated EasyBuild configuration, viz. Python 3.7.4, NumPy 1.18.1, the SciPy Bundle (SciPy 1.4.1, SciKit-Learn 0.22.1, iPython 7.11.1, MatPlotLib 3.1.2, Pandas 0.23.1), and TensorFlow 1.15.2). For additional thrill, there should be two versions of NumPy, one installed with the MKL backend, the other without (using the default, which we believe is OpenBLAS). Ideally, this choice near the bottom of the dependency tree should not propagate into the higher-level modules, i.e. we would hope to have only once instance of the SciPy bundle or TensorFlow, and they would interoperate seamlessly with either choice for NumPy.
Important stuff to remember
export EB_PYTHON=python3
module load EasyBuild/4.3.0
Playground on Saga: /cluster/shared/nlpl/software/easybuild4
export EASYBUILD_ROBOT_PATHS=/cluster/software/EasyBuild/4.3.0/easybuild/easyconfigs:/cluster/shared/nlpl/software/easybuild4
Repository: https://source.coderefinery.org/nlpl/easybuild
References
Compute Canada: https://www.youtube.com/watch?v=_0j5Shuf2uE
Jülich & Ghent: http://easybuilders.github.io/easybuild/files/eb-jsc-hust16.pdf
