Eosc/pretraining/nvidia

Background

This page provides a recipe to large-scale pre-training of a BERT neural language model, using the high-efficiency NVIDIA BERT implementation (which is based on TensorFlow, in contrast to the NVIDIA Megatron code).

Software Installation

Prerequisites

We assume that EasyBuild and Lmod are already installed on the host machine.

We also assume that core software (compilers, most toolchains, CUDA drivers, etc) are also already installed system-wide, or at least that their easyconfigs are available to the system-wide EasyBuild installation.

Finally, the host machine must have Internet connection.

Setting things up

• Clone our repository: git clone https://source.coderefinery.org/nlpl/easybuild.git --branch ak-dev --single-branch
• Its directory ('easybuild') will serve as your building factory. Rename it to whatever you think fits well. Change to this directory.
• To use the same procedure across different systems we provide a custom preparation script.
• To prepare directories and get the path settings run it:

./setdir.sh

• It will create the file SETUP.local with the settings you are going to use in the future (by simply running source SETUP.local after loading EasyBuild).
• It will also print a command for your users to run if they want to be able to load your custom modules.

Building software and installing custom modules

• Load EasyBuild (e.g., module load EasyBuild/4.3.0)
• Load your settings: source SETUP.local
• Check your settings: eb --show-config
• Imitate NVIDIA BERT installation by running

eb --robot NLPL-nvidia_BERT_TF-20.06.08-gomkl-2019b-Python3.7.4.eb --dry-run

• EasyBuild will show the list of required modules, marking those which have to be installed from scratch (by downloading and building the corresponding software).
• If no warning or errors were shown, build everything required by NVIDIA BERT:

eb --robot NLPL-nvidia_BERT_TF-20.06.08-gomkl-2019b-Python3.7.4.eb

• After the process is finished, your modules will be visible along with the system-provided ones via module avail, and can be loaded with module load.
• Approximate building time for all the required modules is several hours (from 3 to 5).

Data Preparation

To train BERT, 3 data pieces are required:

• a training corpus (a collection of plain text files)
• a WordPiece vocabulary (a plain text file)
• a BERT configuration defining the model hyperparameters (a JSON file)

Ready-to-use toy examples data can be found in the tests/text_data subdirectory:

• no_wiki/: a directory with texts from Norwegian Wikipedia (about 1.2 million words)
• norwegian_wordpiece_vocab_20k.txt: Norwegian WordPiece vocabulary (20 000 entries)
• norbert_config.json: BERT configuration file replicating BERT-Small for English (adapted to the number of entries in the vocabulary)