Vectors/elmo/tutorial - Revision history

Andreku: /* Training ELMo on Saga */

2020-02-03T00:12:54Z

‎Training ELMo on Saga

Andreku: /* Using the NLPL environment */

2020-01-31T17:56:15Z

‎Using the NLPL environment

Andreku: /* Training ELMo on Saga */

2020-01-31T17:55:56Z

‎Training ELMo on Saga

Andreku: /* Using Anaconda */

2019-11-15T20:40:45Z

‎Using Anaconda

Andreku: /* Using system TensorFlow */

2019-11-15T20:40:05Z

‎Using system TensorFlow

Andreku: /* Training ELMo on Saga */

2019-11-15T20:39:28Z

‎Training ELMo on Saga

Andreku: /* Using system TensorFlow */

2019-11-15T20:36:31Z

‎Using system TensorFlow

Andreku: /* Training ELMo on Saga */

2019-11-15T20:36:01Z

‎Training ELMo on Saga

Andreku: /* Training ELMo on Saga */

2019-11-15T20:34:30Z

‎Training ELMo on Saga

Andreku: /* Using system TensorFlow */

2019-11-15T19:16:21Z

‎Using system TensorFlow

@@ Line 33: / Line 33: @@
 *using a system ''TensorFlow'' module
 *using ''Anaconda''.
-The speed is comparable:  one epoch over 100 million word tokens takes about 3 hours with 2 GPUs and batch size 192.
+The speed is comparable:  one epoch over 100 million word tokens takes about 3 hours with 2 NVIDIA P100 GPUs and batch size 192.
 == Using the ''NLPL'' environment ==

@@ Line 37: / Line 37: @@
 == Using the ''NLPL'' environment ==
 You will need to load the ''NLPL''-provided modules '''nlpl-python-candy/201912/3.7''' and '''nlpl-tensorflow/1.15.2/3.7'''
 Example SLURM file:
@@ Line 56: / Line 55: @@
    module load nlpl-python-candy/201912/3.7 nlpl-tensorflow/1.15.2/3.7
    python3 bin/train_elmo.py --train_prefix $DATA --size $SIZE  --vocab_file $VOCAB --save_dir $OUT
 == Using system TensorFlow ==

← Older revision		Revision as of 17:55, 31 January 2020
Line 29:		Line 29:


−	There are currently ~~two~~ options of training ELMo on Saga with GPU-enabled ''TensorFlow'': using a system ''TensorFlow'' module~~, or~~ using ''Anaconda''.	+	There are currently three options of training ELMo on Saga with GPU-enabled ''TensorFlow'':
		+	*using the ''NLPL'' environment (recommended)
		+	*using a system ''TensorFlow'' module
		+	*using ''Anaconda''.
	The speed is comparable: one epoch over 100 million word tokens takes about 3 hours with 2 GPUs and batch size 192.		The speed is comparable: one epoch over 100 million word tokens takes about 3 hours with 2 GPUs and batch size 192.
		+
		+	== Using the ''NLPL'' environment ==
		+	You will need to load the ''NLPL''-provided modules '''nlpl-python-candy/201912/3.7''' and '''nlpl-tensorflow/1.15.2/3.7'''
		+
		+
		+	Example SLURM file:
		+
		+	#!/bin/bash
		+	#SBATCH --job-name=ELMo
		+	#SBATCH --mail-type=FAIL
		+	#SBATCH --account=nn9447k # Use your project number
		+	#SBATCH --partition=accel # To use the accelerator nodes
		+	#SBATCH --gres=gpu:2 # To specify how many GPUs to use
		+	#SBATCH --time=10:00:00 # Max walltime is 14 days.
		+	#SBATCH --mem-per-cpu=6G
		+	#SBATCH --ntasks=8
		+	set -o errexit # Recommended for easier debugging
		+	## Load your modules
		+	module purge # Recommended for reproducibility
		+	module use -a /cluster/shared/nlpl/software/modules/etc
		+	module load nlpl-python-candy/201912/3.7 nlpl-tensorflow/1.15.2/3.7
		+	python3 bin/train_elmo.py --train_prefix $DATA --size $SIZE --vocab_file $VOCAB --save_dir $OUT

@@ Line 54: / Line 54: @@
 == Using Anaconda ==
-If using ''Anaconda'', you should install the tensorflow-gpu ''Python'' package locally.
+If using ''Anaconda'', you should install the tensorflow-gpu ''Python'' package locally. The profit is that you can choose (to some extent) the version of ''TensorFlow''.
 Example SLURM file:

@@ Line 34: / Line 34: @@
 == Using system TensorFlow ==
-If using system ''TensorFlow'', you do not have to install it locally.
+If using system ''TensorFlow'' (TensorFlow/1.13.1-fosscuda-2018b-Python-3.6.6), you do not have to install it locally.
 Example SLURM file:

← Older revision		Revision as of 20:39, 15 November 2019
Line 29:		Line 29:


−	There are currently two options of training ELMo on Saga with GPU-enabled ''TensorFlow'': using a system ''TensorFlow'' module, or using ''Anaconda''. ~~We describe both ways~~:	+	There are currently two options of training ELMo on Saga with GPU-enabled ''TensorFlow'': using a system ''TensorFlow'' module, or using ''Anaconda''.
		+	The speed is comparable: one epoch over 100 million word tokens takes about 3 hours with 2 GPUs and batch size 192.
		+

	== Using system TensorFlow ==		== Using system TensorFlow ==

@@ Line 12: / Line 12: @@
 = Training ELMo on Saga =
-There are currently two options of training ELMo on Saga with GPU-enabled ''TensorFlow'': using a system ''TensorFlow'' module, or using ''Anaconda''.
+We recommend to use the code from https://github.com/ltgoslo/simple_elmo_training to train an ELMo model with ''TensorFlow'' .
 After cloning the repository and installing the dependencies, it boils down to running the command
@@ Line 30: / Line 28: @@
 $OUT is a directory where the TensorFlow checkpoints will be saved.
 == Using system TensorFlow ==

@@ Line 14: / Line 14: @@
 There are currently two options of training ELMo on Saga with GPU-enabled ''TensorFlow'': using a system ''TensorFlow'' module, or using ''Anaconda''.
-In both cases, one should use the code from https://github.com/akutuzov/bilm-tf to train a model. It boils down to running the command
+In both cases, one should use the code from https://github.com/ltgoslo/simple_elmo_training to train a model.
   python3 bin/train_elmo.py --train_prefix $DATA --size $SIZE --vocab_file $VOCAB --save_dir $OUT

@@ Line 43: / Line 43: @@
   #SBATCH --mem-per-cpu=6G
   #SBATCH --ntasks=8
   set -o errexit  # Recommended for easier debugging
   ## Load your modules
   module purge   # Recommended for reproducibility