Difference between revisions of "Vectors/elmo/tutorial"

From Nordic Language Processing Laboratory
Jump to: navigation, search
(Training ELMo on Saga)
(Training ELMo on Saga)
Line 12: Line 12:
 
= Training ELMo on Saga =
 
= Training ELMo on Saga =
  
There are currently two options of training ELMo on Saga with GPU-enabled ''TensorFlow'': using a system ''TensorFlow'' module, or using ''Anaconda''.
+
We recommend to use the code from https://github.com/ltgoslo/simple_elmo_training to train an ELMo model with ''TensorFlow'' .  
 
 
In both cases, one should use the code from https://github.com/ltgoslo/simple_elmo_training to train a model.  
 
  
 
After cloning the repository and installing the dependencies, it boils down to running the command
 
After cloning the repository and installing the dependencies, it boils down to running the command
Line 30: Line 28:
 
$OUT is a directory where the TensorFlow checkpoints will be saved.
 
$OUT is a directory where the TensorFlow checkpoints will be saved.
  
 +
 +
There are currently two options of training ELMo on Saga with GPU-enabled ''TensorFlow'': using a system ''TensorFlow'' module, or using ''Anaconda''. We describe both ways:
  
 
== Using system TensorFlow ==
 
== Using system TensorFlow ==

Revision as of 20:36, 15 November 2019

Background

ELMo is a family of contextualized word embeddings first introduced in [Peter et al. 2018].

Using pre-trained models

Pre-trained ELMo models are available from the NLPL Word Embeddings repository.

Python code to infer contextualized word vectors from any input text, given a pre-trained model:

https://github.com/ltgoslo/simple_elmo

Training ELMo on Saga

We recommend to use the code from https://github.com/ltgoslo/simple_elmo_training to train an ELMo model with TensorFlow .

After cloning the repository and installing the dependencies, it boils down to running the command

python3 bin/train_elmo.py --train_prefix $DATA --size $SIZE --vocab_file $VOCAB --save_dir $OUT

where

$DATA is a path to the directory containing any number of (possibly gzipped) plain text files: your training corpus.

$SIZE if the number of word tokens in $DATA (necessary to properly construct and log batches).

$VOCAB is a (possibly gzipped) one-word-per-line vocabulary file to be used for language modeling; it should always contain at least <S>, </S> and <UNK>.

$OUT is a directory where the TensorFlow checkpoints will be saved.


There are currently two options of training ELMo on Saga with GPU-enabled TensorFlow: using a system TensorFlow module, or using Anaconda. We describe both ways:

Using system TensorFlow

If using system TensorFlow, you do not have to install any Python modules yourself.

Example SLURM file:

#!/bin/bash
#SBATCH --job-name=elmo
#SBATCH --mail-type=FAIL
#SBATCH --account=nn9447k    # Use your project number
#SBATCH --partition=accel    # To use the accelerator nodes
#SBATCH --gres=gpu:2         # To specify how many GPUs to use
#SBATCH --time=10:00:00      # Max walltime is 14 days.
#SBATCH --mem-per-cpu=6G
#SBATCH --ntasks=8
set -o errexit  # Recommended for easier debugging
## Load your modules
module purge   # Recommended for reproducibility
module load TensorFlow/1.13.1-fosscuda-2018b-Python-3.6.6
python3 bin/train_elmo.py --train_prefix $DATA --size $SIZE  --vocab_file $VOCAB --save_dir $OUT

Using Anaconda

If using Anaconda, you should install the tensorflow-gpu Python package locally.

Example SLURM file:

#!/bin/bash
#SBATCH --job-name=elmo
#SBATCH --mail-type=FAIL
#SBATCH --account=nn9447k  # Use your project number
#SBATCH --partition=accel    # To use the accelerator nodes
#SBATCH --gres=gpu:2         # To specify how many GPUs to use
#SBATCH --time=10:00:00      # Max walltime is 14 days.
#SBATCH --mem-per-cpu=6G
#SBATCH --ntasks=8
set -o errexit  # Recommended for easier debugging
module purge   # Recommended for reproducibility
module load Anaconda3/2019.03
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/cluster/software/Anaconda3/2019.03/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
   eval "$__conda_setup"
else
   if [ -f "/cluster/software/Anaconda3/2019.03/etc/profile.d/conda.sh" ]; then
               . "/cluster/software/Anaconda3/2019.03/etc/profile.d/conda.sh"
   else
       export PATH="/cluster/software/Anaconda3/2019.03/bin:$PATH"
   fi
fi
unset __conda_setup
# <<< conda initialize <<<
conda activate python3.6
python3 bin/train_elmo.py --train_prefix $DATA --size $SIZE --vocab_file $VOCAB --save_dir $OUT