Eosc/horovod - Revision history

Oe: /* The Inevitable: Backgrading */

2020-02-01T23:03:26Z

‎The Inevitable: Backgrading

Oe: /* Four Weeks Later: Performance Optimization */

2020-02-01T23:02:57Z

‎Four Weeks Later: Performance Optimization

Oe: /* High-Level Goal: Multi-GPU Training */

2020-02-01T23:01:30Z

‎High-Level Goal: Multi-GPU Training

Oe: /* Questions: Installation and Deployment */

2020-02-01T22:29:11Z

‎Questions: Installation and Deployment

Oe: /* Questions: Installation and Deployment */

2020-02-01T22:28:57Z

‎Questions: Installation and Deployment

Oe: /* Reflections: Responsibilities */

2020-02-01T22:12:19Z

‎Reflections: Responsibilities

Oe: /* Reflections: Responsibilities */

2020-02-01T22:11:30Z

‎Reflections: Responsibilities

Oe: /* Four Weeks Later: Performance Optimization */

2020-01-31T15:15:55Z

‎Four Weeks Later: Performance Optimization

Oe: /* Four Weeks Later: Performance Optimization */

2020-01-30T22:26:16Z

‎Four Weeks Later: Performance Optimization

Oe: /* Four Weeks Later: Performance Optimization */

2020-01-30T22:25:15Z

‎Four Weeks Later: Performance Optimization

@@ Line 57: / Line 57: @@
 code running in TensorFlow 1.15 and Python 3.7 (for reasons beyond their control).
 They ask for an installation of TensorFlow 1.12 in a Python 3.5 environment.
-They can make do without Horovod, but they badly need Gensim and spaCy.
+They can maybe make do without Horovod, but they badly need Gensim and spaCy.
 = Reflections: Responsibilities =

@@ Line 48: / Line 48: @@
 From colleagues in Finland, they understand that running Horovod on top of the
 Intel MPI Library (instead of OpenMPI) can give a performance gain of up to 40 percent.
-Reliable Intel MPI utilization, however, requires installation of the most recent
+Reliable Intel MPI support, however, requires installation of the most recent
 Horovod 0.20.0 release candidate (while keeping TensorFlow, Gensim, spaCy, et al.
 versions as before).

@@ Line 20: / Line 20: @@
 users to activate and run with minimal effort: Horovod 0.19.0, with either PyTorch 1.4.0 or
 TensorFlow 1.15.2 as its deep learning (DL) backend, all in a Python 3.7.2 environment.
-Horovod has a number of dependencies, including a basic tool chain, NCCL 2 and a suitable MPI
+Horovod has a number of dependencies, including a basic C++ tool chain, NCCL 2, and a suitable MPI
 implementation (e.g. OpenMPI 3.1.2 or 4.0.0, but not 3.1.3).
 The DL backends each bring their own set of dependencies, currently among other things

@@ Line 100: / Line 100: @@
 tools and assist others in using them effectively.
-= Questions: Installation and Deployment =
+= Questions: Installation vs. Deployment =
 Provisioning the above software stack on multiple HPC systems (e.g. the Puhti and Saga

← Older revision		Revision as of 22:28, 1 February 2020
Line 101:		Line 101:

	= Questions: Installation and Deployment =		= Questions: Installation and Deployment =
		+
		+	Provisioning the above software stack on multiple HPC systems (e.g. the Puhti and Saga
		+	superclusters) raises question of modularization, automated installation, package management,
		+	portability, and replicability.
		+	So far, project partners have discussed three broad types of technologies:
		+	(a) automated compilation, building on a common foundation of core components,
		+	using frameworks like [https://easybuild.readthedocs.io/en/latest/ EasyBuild]
		+	or [https://spack.io/ Spack];
		+	(b) a package manager and repository, e.g. establishing something like
		+	an NLPConda channel; and
		+	(c) containerization, e.g. using Singularity images for portable deployment
		+	across different HPC (and possibly also cloud) systems.
		+
		+	A full solution will possibly require more than one of these technologies.
		+	While attractive for portability, it is not immediately clear how container
		+	construction and management can support the fine-grained modularization and
		+	‘mixing and matching’ of components under user control required in the above
		+	Horovod scenario.
		+	Containers in and of themselves do not address the build and installation
		+	requirements, and in early 2020 at least it is unclear what levels of
		+	support are available on the current superclusters, and what to expect
		+	on future systems (notably the LUMI environment).
		+	For these reasons, containerization appears more like a candidate mid-term
		+	deployment vehicle than the primary solution for software maintenance per se.

← Older revision		Revision as of 22:12, 1 February 2020
Line 99:		Line 99:
	laboratory: expert members of the community are best positioned to install these		laboratory: expert members of the community are best positioned to install these
	tools and assist others in using them effectively.		tools and assist others in using them effectively.
		+
		+	= Questions: Installation and Deployment =

@@ Line 60: / Line 60: @@
 = Reflections: Responsibilities =
 Core components like the C++ tool chain, CUDA and associated extensions, different
@@ Line 68: / Line 75: @@
 disciplines or user groups.
-A similar argument could be made for general-purpose DL frameworks like
+Although less intricately tied to the host environment, general-purpose DL frameworks
-PyTorch and TensorFlow, but
+are not discipline-specific either and could in principle be considered part of the

@@ Line 45: / Line 45: @@
 A team of NLPL developers sets out to adapt for Norwegian a large-scale experiment
 originally published by Google AI researchers, who report that they cumulatively
-used two TPU years (two weaks with exclusive access to 64 units) on this computation.
+used two TPU years (two weeks of exclusive access to 64 units) on this computation.
 From colleagues in Finland, they understand that running Horovod on top of the
 Intel MPI Library (instead of OpenMPI) can give a performance gain of up to 40 percent.