Eosc/pretraining/nvidia

Background

This page provides a recipe to large-scale pre-training of a BERT neural language model, using the high-efficiency NVIDIA BERT implementation (which is based on TensorFlow, in contrast to the NVIDIA Megatron code).

Software Installation

Prerequisites

We assume that EasyBuild and Lmod are already installed on the host machine.

We also assume that core software (compilers, most toolchains, CUDA drivers, etc) are also already installed system-wide, or at least that their easyconfigs are available to the system-wide EasyBuild installation.

Finally, the host machine must have Internet connection.

Setting things up

• Clone our repository: git clone https://source.coderefinery.org/nlpl/easybuild.git --branch ak-dev --single-branch
• Its directory ('easybuild') will serve as your building factory. Rename it to whatever you think fits well. Change to this directory.
• To use the same procedure across different systems we provide a custom preparation script.
• To prepare directories and get the path settings run it:

./setdir.sh

• It will create the file SETUP.local with the settings you are going to use in the future (by simply running source SETUP.local after loading EasyBuild).
• It will also print a command for your users to run if they want to be able to load your custom modules.
• Once you have created your modules (see below), they will be visible along with the system-provided ones via module avail, and can be loaded with module load.