Difference between revisions of "Infrastructure/software/python"

From Nordic Language Processing Laboratory
Jump to: navigation, search
(Created page with "= Background = In mid-2018 at least, the Python programming language takes a central role in much data science and machine learning work. Comprehensive programming environmen...")
 
(Background)
Line 8: Line 8:
 
[https://www.tensorflow.org/ TensorFlow] often make
 
[https://www.tensorflow.org/ TensorFlow] often make
 
it possible to deploy advanced machine learning techniques
 
it possible to deploy advanced machine learning techniques
with a few lines of glue code in Python.
+
with a few lines of ‘glue’ code in Python.
 
The Python ecosystem is characterized by high degrees
 
The Python ecosystem is characterized by high degrees
 
of evolution and fragmentation:
 
of evolution and fragmentation:

Revision as of 13:48, 2 October 2018

Background

In mid-2018 at least, the Python programming language takes a central role in much data science and machine learning work. Comprehensive programming environments like SciKitLearn, PyTorch, or TensorFlow often make it possible to deploy advanced machine learning techniques with a few lines of ‘glue’ code in Python. The Python ecosystem is characterized by high degrees of evolution and fragmentation: The core language still develops, there are two mutually incompatible major versions still in common use, and even inbetween minor versions there have at times been incompatible changes. On top of this dynamically evolving core, there is a vast landscape of independently developed add-on modules, of which some are near-universally used (e.g. NumPy or MatPlotLib. For these reasons, maintaining a Python environment for NLPL that is to some degree standardized and parallel across the various instances of the virtual laboratory is no small challenge.

Basic Principles

NLPL standardizes on Python 3.x. In mid-2018, we use Python version 3.5 as the default starting point for NLPL-specific installations of add-on modules.


Vague Thoughts

In principle, one could reduce most of the virtual environments to just the wheels that they provide, i.e. remove everything else from the standard virtual environment creation, including the binaries. As the wheels are organized in a directory hierarchy relative to each Python minor version, this should make it possible to install the code for add-on components (like MatPlotLib, GenSim, or NLTK) for multiple Python versions into the same module.