Infrastructure/software/nltk

From Nordic Language Processing Laboratory
(Difference between revisions)
Jump to: navigation, search
(Installation on Abel)
(Usage on Abel)
Line 47: Line 47:
 
python is /projects/nlpl/software/tensorflow/1.11/bin/python
 
python is /projects/nlpl/software/tensorflow/1.11/bin/python
 
python is /projects/nlpl/software/nltk/3.3/bin/python
 
python is /projects/nlpl/software/nltk/3.3/bin/python
 +
python is /usr/bin/python
 +
python is /opt/rocks/bin/python
 +
</pre>
 +
 +
= Usage on Taito =
 +
 +
The module <tt>nlpl-nltk</tt> provides an NLTK installation
 +
in a Python 3.5 virtual environment.
 +
 +
<pre>
 +
module purge
 +
module use -a /proj/nlpl/software/modulefiles
 +
module load nlpl-nltk
 +
</pre>
 +
 +
This installation (just as other NLPL-maintained Python virtual environments)
 +
can be combined with other Python-based modules, for example the NLPL
 +
installations of PyTorch or OpenNMT-py.
 +
To ‘stack’ multiple Python environments, they can simply be loaded together,
 +
e.g.
 +
 +
<pre>
 +
module load nlpl-nltk nlpl-pytorch
 +
</pre>
 +
 +
Because PyTorch is somewhat ‘special’ in its requirements for
 +
dynamic libraries and support for both cpu and gpu nodes, it is important
 +
for it to be activated last, i.e. on the ‘top’ of a multi-module stack.
 +
This can be validated by inspecting which <tt>python</tt> binary is active
 +
according to the search order in the <tt>$PATH</tt> environment variable:
 +
<pre>
 +
type -all python
 +
</pre>
 +
 +
In late September 2018, for example, the output from the above command
 +
would look as follows:
 +
<pre>
 +
python is /proj/nlpl/software/pytorch/0.4.1/bin/python
 +
python is /proj/nlpl/software/nltk/3.3/bin/python
 
python is /usr/bin/python
 
python is /usr/bin/python
 
python is /opt/rocks/bin/python
 
python is /opt/rocks/bin/python

Revision as of 21:25, 29 September 2018

Contents

Background

The Natural Language Toolkit (NLTK) provides a large collection of core NLP utilities (e.g. sentence splitting and tokenization, part of speech tagging, various approaches to parsing, and many more) in an integrated Python environment. The NLTK distribution also bundles a broad range of common, freely available data sets, which are made accessible through a uniform API. Albeit often neither quite state of the art nor blindingly efficient, NLTK is popular as a teaching environment and go-to repository of common ‘basic’ preprocessing tasks, e.g. sentence splitting, stop word removal, or lemmatization (for English, at least).

Usage on Abel

The module nlpl-nltk provides an NLTK installation in a Python 3.5 virtual environment.

module purge
module use -a /projects/nlpl/software/modulefiles
module load nlpl-nltk

This installation (just as other NLPL-maintained Python virtual environments) can be combined with other Python-based modules, for example the NLPL installations of PyTorch or TensorFlow. To ‘stack’ multiple Python environments, they can simply be loaded together, e.g.

module load nlpl-nltk nlpl-tensorflow

Because PyTorch and TensorFlow are ‘special’ in their requirements for dynamic libraries and support for both cpu and gpu nodes, it is important for them to be activated last, i.e. on the ‘top’ of a multi-module stack. This can be validated by inspecting which python binary is active according to the search order in the $PATH environment variable:

type -all python

In late September 2018, for example, the output from the above command would look as follows:

python is /projects/nlpl/software/tensorflow/1.11/bin/python
python is /projects/nlpl/software/nltk/3.3/bin/python
python is /usr/bin/python
python is /opt/rocks/bin/python

Usage on Taito

The module nlpl-nltk provides an NLTK installation in a Python 3.5 virtual environment.

module purge
module use -a /proj/nlpl/software/modulefiles
module load nlpl-nltk

This installation (just as other NLPL-maintained Python virtual environments) can be combined with other Python-based modules, for example the NLPL installations of PyTorch or OpenNMT-py. To ‘stack’ multiple Python environments, they can simply be loaded together, e.g.

module load nlpl-nltk nlpl-pytorch

Because PyTorch is somewhat ‘special’ in its requirements for dynamic libraries and support for both cpu and gpu nodes, it is important for it to be activated last, i.e. on the ‘top’ of a multi-module stack. This can be validated by inspecting which python binary is active according to the search order in the $PATH environment variable:

type -all python

In late September 2018, for example, the output from the above command would look as follows:

python is /proj/nlpl/software/pytorch/0.4.1/bin/python
python is /proj/nlpl/software/nltk/3.3/bin/python
python is /usr/bin/python
python is /opt/rocks/bin/python

Available Versions

Installation on Abel or Taito

module purge
module load python3/3.5.0
mkdir ${NLPLROOT}/software/nltk
virtualenv ${NLPLROOT}/software/nltk/3.3

Next, we need to create a module definition, in this case ${NLPLROOT}/software/modulefiles/nlpl-nltk/3.3 (on Abel) or ${NLPLROOT}/software/modulefiles/nlpl-nltk/3.3.lua (on Taito); make sure to establish the environment variable $NLTK_DATA, pointing to the data sub-directory of the NLTK tree, as established by the command-line data download below.

module load nlpl-nltk/3.3
pip install --upgrade pip
pip install --upgrade $(pip list | tail -n +3 | gawk '{print $1}')

Finally, install the NLTK code and all data packages.

pip install nltk
python -m nltk.downloader -d ${NLPLROOT}/software/nltk/3.3/data all
Personal tools
Namespaces

Variants
Actions
Navigation
Tools