Eosc/pretraining/nvidia - Revision history

Andreku: /* Distributed filesystems */

2021-07-07T16:26:00Z

‎Distributed filesystems

Andreku: /* Distributed filesystems */

2021-05-10T15:09:02Z

‎Distributed filesystems

Andreku: /* Training Example */

2021-04-23T20:57:12Z

‎Training Example

Andreku: /* Testing */

2021-04-23T20:55:08Z

‎Testing

Andreku at 20:52, 23 April 2021

2021-04-23T20:52:40Z

Andreku: /* Fine-tuning */

2021-04-23T20:50:12Z

‎Fine-tuning

Andreku: /* Software Installation */

2021-04-23T20:48:59Z

‎Software Installation

Andreku: /* Setting things up */

2021-04-23T20:44:36Z

‎Setting things up

Andreku: /* Setting things up */

2021-04-23T20:44:24Z

‎Setting things up

Andreku: /* IMPORTANT NOTE FOR NVIDIA A100 GPUs */

2021-04-23T20:41:41Z

‎IMPORTANT NOTE FOR NVIDIA A100 GPUs

← Older revision		Revision as of 16:26, 7 July 2021
Line 119:		Line 119:

	After this, GCC must be rebuilt.		After this, GCC must be rebuilt.
		+
		+	Another solution is to add the following line to the binutils easyconfig ('''binutils-2.32-GCCcore-8.3.0.eb'''):
		+
		+	'''configopts = ' --enable-gold=no''''

	== Binary blobs ==		== Binary blobs ==

@@ Line 112: / Line 112: @@
 For example, on Puhti cluster, one can use $TMPDIR (this is a local ext4 partition).
 When building on a compute node, this should be set to $LOCAL_SCRATCH.
 == Binary blobs ==

@@ Line 80: / Line 80: @@
 = Training Example =
 * Extend your $MODULEPATH with the path to your custom modules, using the command suggested by the '''setdir.sh''' script:
-* '''module use -a PATH_TO_YOUR_REPOSITORY/easybuild/install/modules/all/'''
+* '''module use -a PATH_TO_YOUR_REPOSITORY/modules/all/'''
 * Load the NVIDIA BERT module:
 * '''module load nlpl-nvidia-bert/20.06.8-gomkl-2019b-tensorflow-1.15.2-Python-3.7.4'''

← Older revision		Revision as of 20:55, 23 April 2021
Line 99:		Line 99:
	It will tell you whether TensorFlow sees your GPU and will run simple calculations.		It will tell you whether TensorFlow sees your GPU and will run simple calculations.
	The execution time should not be more than 10 minutes (usually much less).		The execution time should not be more than 10 minutes (usually much less).
		+
		+	= Possible building issues =
		+
		+	== Too restrictive login nodes ==
		+	On most HPC clusters, our modules can be built on login nodes.
		+	However, in some systems, login nodes might be too restrictive in terms of maximum thread number or memory limit.
		+	In this case, we recommend to build heavy modules (Java, Bazel, TensorFlow, PyTorch) on compute nodes.
		+
		+	== Distributed filesystems ==
		+	There is a [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 known problem] with building some software (GCC, Java) on a distributed file system like Lustre.
		+	If you hit this, simply change the EasyBuild build directory to any local path.
		+	For example, on Puhti cluster, one can use $TMPDIR (this is a local ext4 partition).
		+	When building on a compute node, this should be set to $LOCAL_SCRATCH.
		+
		+	== Binary blobs ==
		+	Most modules in our virtual laboratory will be built in a completely unsupervised way, and all the necessary source archives for them will be downloaded automatically.
		+	However, some software packages require manually downloading binary blobs from their websites after registration.
		+	Our scripts can't do that for you. Make sure to put the downloaded archives into the ''blobs'' subdirectory. Some of the examples include:
		+
		+	* '''jdk-8u212-linux-x64.tar.gz''' is required to build Java to build Bazel (for TensorFlow),
		+	* '''cudnn-10.1-linux-x64-v7.6.4.38.tgz''' is required to build cuDNN (for TensorFlow),
		+	* '''nccl_2.6.4-1+cuda10.1_x86_64.txz''' is required to build NCCL (for TensorFlow).
		+
		+	Your system might already provide the necessary versions of Java, cuDNN and NCCL as modules. In this case, you do not have to download anything.

← Older revision		Revision as of 20:52, 23 April 2021
Line 89:		Line 89:
	* Training on the toy data above takes not more than an hour on 4 Saga GPUs.		* Training on the toy data above takes not more than an hour on 4 Saga GPUs.
	* We use 4 GPUs by default (to test hardware and software as much as possible). Modify the '''train_bert.sh''' script to change this or other BERT training parameters.		* We use 4 GPUs by default (to test hardware and software as much as possible). Modify the '''train_bert.sh''' script to change this or other BERT training parameters.
		+
		+	= Testing =
		+	It is important to make sure that Numpy is built with accelerated linear algebra libraries (OpenBLAS or MKL).
		+	To test this, we provide the [https://source.coderefinery.org/nlpl/easybuild/-/blob/master/tests/numpy/numpy_test.py numpy_test.py] script.
		+	Its execution time with our ''nlpl-numpy*'' modules should not be more than 10 minutes (usually much less).
		+
		+	It is also important to make sure that TensorFlow is built with proper GPU support.
		+	To test this, we provide the [https://source.coderefinery.org/nlpl/easybuild/-/blob/master/tests/tensor/tensorflow_test.py tensorflow_test.py] script.
		+	It will tell you whether TensorFlow sees your GPU and will run simple calculations.
		+	The execution time should not be more than 10 minutes (usually much less).

@@ Line 63: / Line 63: @@
 By default, our easyconfigs perform building with levels 6.0 and 7.0.
 But if your system uses only one particular type of GPUs (which is most probably the case), you should choose only the corresponding level, to decrease the building time and the resulting binaries size.
-Simply choose the level suitable for your GPUs and change this line in the TensorFlow or PyTorch easyconfigs to your choice:
+Simply choose the level suitable for your GPUs and change this line in the [https://source.coderefinery.org/nlpl/easybuild/-/blob/master/easyconfigs/nlpl-tensorflow-1.15.2-foss-2019b-cuda-10.1.243-Python-3.7.4.eb TensorFlow] or [https://source.coderefinery.org/nlpl/easybuild/-/blob/master/easyconfigs/nlpl-pytorch-1.6.0-foss-2019b-cuda-10.1.243-Python-3.7.4.eb PyTorch] easyconfigs to your choice:
 '''cuda_compute_capabilities = ["6.0", "7.0"]'''

@@ Line 21: / Line 21: @@
 We also assume that core software (compilers, most toolchains, CUDA drivers, etc) are already installed system-wide as well,
-or at least that their [https://github.com/easybuilders/easybuild-easyconfigs easyconfigs] are available to the system-wide EasyBuild installation.
+or at least that their [https://github.com/easybuilders/easybuild-easyconfigs easyconfigs] (module description files) are available to the system-wide EasyBuild installation.
 If your system already provides exactly the same version of some software package that we need, we will use it. If not, it will be built from scratch.
@@ Line 58: / Line 58: @@
 * After the process is finished, your modules will be visible along with the system-provided ones via '''module avail''', and can be loaded with '''module load'''.
 * Approximate building time for all the required modules is several hours (from 3 to 5).
 = Data Preparation =

@@ Line 41: / Line 41: @@
 * To prepare directories and get the path settings, run it: '''./setdir.sh'''
 * If your EasyBuild version is different from 4.3.3, change the version in the '''setdir.sh''' script.
-* By default, your custom modules will be stored in the ''modules'' subdirectory, and the compiled software itself will be stored in the ''software'' subdirectory.
+* By default, your custom modules will be stored in the ''modules'' subdirectory, and the compiled software itself will be stored in the ''software'' subdirectory. If needed, you can change the '''EASYBUILD_INSTALLPATH_MODULES''' and '''EASYBUILD_INSTALLPATH_SOFTWARE''' variables in the '''setdir.sh''' script.
 * The script will create the '''SETUP.local''' file with the settings you are going to use in the future (by simply running '''source SETUP.local''' after loading EasyBuild).
 * It will also print a command for your users to run if they want to be able to load your custom modules ('''module use -a PATH-TO-MODULES'''). You can add it to your '''.bashrc''' file, or put it in all your SLURM scripts.

@@ Line 39: / Line 39: @@
 * Its directory ('easybuild') will serve as your building factory. Rename it to whatever you think fits well. Change to this directory.
 * To use the same procedure across different systems we provide a custom preparation script.
-* To prepare directories and get the path settings run it:
+* To prepare directories and get the path settings, run it: '''./setdir.sh'''
-'''./setdir.sh'''
+* If your EasyBuild version is different from 4.3.3, change the version in the '''setdir.sh''' script.
-* It will create the file SETUP.local with the settings you are going to use in the future (by simply running '''source SETUP.local''' after loading EasyBuild).
+* By default, your custom modules will be stored in the ''modules'' subdirectory, and the compiled software itself will be stored in the ''software'' subdirectory.
-* It will also print a command for your users to run if they want to be able to load your custom modules.
+If needed, you can change the '''EASYBUILD_INSTALLPATH_MODULES''' and '''EASYBUILD_INSTALLPATH_SOFTWARE''' variables in the '''setdir.sh''' script.
 ==Building software and installing custom modules ==

@@ Line 26: / Line 26: @@
 Finally, the host machine must have Internet connection.
-===IMPORTANT NOTE FOR NVIDIA A100 GPUs===
+===Important note for NVIDIA A100 GPUs===
 NVIDIA A100 GPUs (any GPUs with CUDA compute capability 8) are designed to work best with CUDA 11 and cuDNN 8. Applications built with CUDA 10 and cuDNN 7 [https://docs.nvidia.com/cuda/ampere-compatibility-guide/index.html in principle can be run on A100] (by JIT-compiling PTX code into GPU binary code on every run), but this is not recommended by NVIDIA.