Eosc/easybuild/andreku - Revision history

Andreku: /* Deployment on Puhti */

2021-04-15T18:55:11Z

‎Deployment on Puhti

Andreku: /* Deployment on Puhti */

2021-03-14T03:16:40Z

‎Deployment on Puhti

Andreku: /* Deployment on Puhti */

2021-03-13T00:37:17Z

‎Deployment on Puhti

Andreku: /* Deployment on Puhti */

2021-03-13T00:16:30Z

‎Deployment on Puhti

Andreku: /* Deployment on Puhti */

2021-03-12T19:49:32Z

‎Deployment on Puhti

Andreku: /* Deployment on Puhti */

2021-03-11T16:45:30Z

‎Deployment on Puhti

Andreku: /* Deployment on Puhti */

2021-03-11T15:42:06Z

‎Deployment on Puhti

Andreku at 15:22, 11 March 2021

2021-03-11T15:22:55Z

Andreku: /* Remaining issues */

2021-01-28T13:18:00Z

‎Remaining issues

Andreku: /* Important stuff to remember */

2021-01-05T12:58:42Z

‎Important stuff to remember

@@ Line 61: / Line 61: @@
 There was an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball. A more general solution (applicable not only to GCC but also to Java, and other packages as well) is to change the EasyBuild build directory to any location not on Lustre distributed file system. One can simply use '''$TMPDIR''' (this is a local ext4 partition).
-Bazel and TensorFlow must be built on a compute node; it seems that the Puhti login nodes are too restrictive in terms of maximum thread number. In this case, the build directory should be set to $LOCAL_SCRATCH:
+Bazel, TensorFlow and PyTorch must be built on a compute node; it seems that the Puhti login nodes are too restrictive in terms of maximum thread number. In this case, the build directory should be set to $LOCAL_SCRATCH:
 '''export EASYBUILD_BUILDPATH=$LOCAL_SCRATCH'''

@@ Line 47: / Line 47: @@
 Initial setup:
 * '''module use -a /projappl/nlpl/software/andreku/modules/all'''
 * '''module load EasyBuild'''

@@ Line 55: / Line 55: @@
 # cudnn-10.1-linux-x64-v7.6.4.38.tgz to build cuDNN to build TensorFlow,
 # nccl_2.6.4-1+cuda10.1_x86_64.txz to build NCCL to build TensorFlow.
 There was an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball. A more general solution (applicable not only to GCC but also to Java, and other packages as well) is to change the EasyBuild build directory to any location not on Lustre distributed file system. One can simply use '''$TMPDIR''' (this is a local ext4 partition).

@@ Line 51: / Line 51: @@
 * '''source SETUP.local'''
-We are building everything from scratch. We borrow the necessary binary blobs (for example the JDK tarball) from Saga, they are stored on Puhti in /projappl/nlpl/software/andreku/extdownloads
+We are building everything from scratch. We borrow the necessary binary blobs from Saga, they are stored on Puhti in /projappl/nlpl/software/andreku/extdownloads. In fact, only three such tarballs are needed to eventually build the NVIDIA BERT implementation:
-There is an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball. A more general solution (applicable not only to Java, but to other packages as well) is to change the EasyBuild build directory to any location not on Lustre distributed file system. I simply use '''/tmp/andreku/''', which is local ext4.
+There was an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball. A more general solution (applicable not only to GCC but also to Java, and other packages as well) is to change the EasyBuild build directory to any location not on Lustre distributed file system. One can simply use '''$TMPDIR''' (this is a local ext4 partition).
-Bazel and TensorFlow must be built on a compute node; it seems that the Puhti login nodes are too restrictive in terms of maximum thread number.
+Bazel and TensorFlow must be built on a compute node; it seems that the Puhti login nodes are too restrictive in terms of maximum thread number. In this case, the build directory should be set to $LOCAL_SCRATCH:

← Older revision		Revision as of 19:49, 12 March 2021
Line 55:		Line 55:
	There is an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball. A more general solution (applicable not only to Java, but to other packages as well) is to change the EasyBuild build directory to any location not on Lustre distributed file system. I simply use '''/tmp/andreku/''', which is local ext4.		There is an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball. A more general solution (applicable not only to Java, but to other packages as well) is to change the EasyBuild build directory to any location not on Lustre distributed file system. I simply use '''/tmp/andreku/''', which is local ext4.

−	Bazel must be built on a compute node; it seems that the Puhti login nodes are too restrictive in terms of maximum thread number.	+	Bazel and TensorFlow must be built on a compute node; it seems that the Puhti login nodes are too restrictive in terms of maximum thread number.

← Older revision		Revision as of 15:42, 11 March 2021
Line 51:		Line 51:
	* '''source SETUP.local'''		* '''source SETUP.local'''

−	We are building everything from scratch. There is an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball.	+	We are building everything from scratch.
		+
		+	There is an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball. A more general solution (applicable not only to Java, but to other packages as well) is to change the EasyBuild build directory to any location not on Lustre distributed file system. I simply use '''/tmp/andreku/''', which is local ext4.

← Older revision		Revision as of 15:22, 11 March 2021
Line 41:		Line 41:
	= Remaining issues =		= Remaining issues =
	* TensorFlow is built with CUDA 10.1.243, not CUDA 10.0.130. Attempts to use the latter [https://source.coderefinery.org/nlpl/easybuild/-/issues/9 failed]. Should find a way to make EasyBuild look for a non-standard CUDA location.		* TensorFlow is built with CUDA 10.1.243, not CUDA 10.0.130. Attempts to use the latter [https://source.coderefinery.org/nlpl/easybuild/-/issues/9 failed]. Should find a way to make EasyBuild look for a non-standard CUDA location.
		+
		+	= Deployment on Puhti =
		+	Location: /projappl/nlpl/software/andreku
		+
		+	Initial setup:
		+
		+	* '''module use -a /projappl/nlpl/software/andreku/modules/all'''
		+	* '''module load EasyBuild'''
		+	* '''source SETUP.local'''
		+
		+	We are building everything from scratch. There is an [https://github.com/easybuilders/easybuild-easyconfigs/issues/12321 issue with GCC building], fixed by manual intervention in the MPC tarball.

← Older revision		Revision as of 13:18, 28 January 2021
Line 41:		Line 41:
	= Remaining issues =		= Remaining issues =
	* TensorFlow is built with CUDA 10.1.243, not CUDA 10.0.130. Attempts to use the latter [https://source.coderefinery.org/nlpl/easybuild/-/issues/9 failed]. Should find a way to make EasyBuild look for a non-standard CUDA location.		* TensorFlow is built with CUDA 10.1.243, not CUDA 10.0.130. Attempts to use the latter [https://source.coderefinery.org/nlpl/easybuild/-/issues/9 failed]. Should find a way to make EasyBuild look for a non-standard CUDA location.
−	* Add easyconfigs for TensorFlow 2.0 and Transformers

@@ Line 9: / Line 9: @@
 Real module location: /cluster/projects/nn9851k/software/easybuild/install/modules/all/
-'''export EASYBUILD_ROBOT_PATHS=/cluster/software/EasyBuild/4.3.0/easybuild/easyconfigs:/cluster/shared/nlpl/software/easybuild_ak'''
+'''export EASYBUILD_ROBOT_PATHS=/cluster/software/EasyBuild/4.3.0/easybuild/easyconfigs:/cluster/projects/nn9851k/software/'''
 (or just '''source PATH.local''')