Difference between revisions of "Infrastructure/software/frameworks"
(→Taito) |
(→Abel) |
||
| Line 29: | Line 29: | ||
= Abel = | = Abel = | ||
| + | In early 2018, the two system-wide installations of DL frameworks | ||
| + | on Abel appear to reflect requests from NLPL users. | ||
| + | |||
| + | [http://www.uio.no/english/services/it/research/hpc/abel/help/software/dynet.html DyNet] no version number specified, but apparently 2.0 (<tt>dynet.__version__</tt>); may be CPU-only; Singularity container | ||
| + | |||
| + | [http://www.uio.no/english/services/it/research/hpc/abel/help/software/tensorflow.html TensorFlow] no version number specified, but apparently 1.0.1 (<tt>tensorflow.__version__</tt>); supports both CPU and GPU nodes; Singularity container. | ||
| + | |||
| + | Installation inside of Singularity containers on Abel limits | ||
| + | interoperability, as there is no straightforward way of ‘mixing and matching’ | ||
| + | with other modules. | ||
= Vision = | = Vision = | ||
Revision as of 14:22, 6 March 2018
Contents
Background
General programming environments for so-called neural or deep learning (DL) are a prerequisite to much current NLP research, somewhat analogous to compilers or core libraries (like Boost or ICU). These frameworks are not in principle discipline-specific and will often be difficult to install in user space (i.e. without administrator privileges). Thus, general DL support should be provided at the system level (not on a per-project basis), and NLPL will seek to work with the system administrators to make sure that (a) relevant frameworks are available (in all relevant versions, both for CPU and GPU usage) and (b) that system-wide installations provide at least some degree of parallelism between the systems available to NLPL users. These are no small ambitions :-).
Taito
It appears that CSC has started to actively support deep learning, possibly at least in part in response to requests from NLPL users. In early 2018, the following packages are available on Taito (although some do not seem easily discoverable via the module command):
DyNet 2.0 and 2.0.1, apparently both for CPU and GPU nodes
TensorFlow many versions, from 0.11 through 1.5.0; seemingly only installed for GPU nodes, though page provides instructions for user-level installation of CPU-only version.
ML Bundle Python 2.x and 3.x installations with multiple DL frameworks included, notably MxNet, PyTorch, Theano, and TensorFlow. No version information available for individual DL frameworks, and version scheme for the bundle at large seems to follow Python versions. Seemingly only available on GPU nodes. Potentially troubling announcement that “packages in the Mlpython environments are updated periodically.”
Abel
In early 2018, the two system-wide installations of DL frameworks on Abel appear to reflect requests from NLPL users.
DyNet no version number specified, but apparently 2.0 (dynet.__version__); may be CPU-only; Singularity container
TensorFlow no version number specified, but apparently 1.0.1 (tensorflow.__version__); supports both CPU and GPU nodes; Singularity container.
Installation inside of Singularity containers on Abel limits interoperability, as there is no straightforward way of ‘mixing and matching’ with other modules.
