Eosc/pretraining

From Nordic Language Processing Laboratory
Revision as of 17:10, 31 August 2020 by Andreku (talk | contribs) (ELMo)
Jump to: navigation, search

Background

This page provides an informal, technically-oriented survey over available (and commonly used) architectures and implementations for large-scale pre-training (and fine-tuning) of contextualized neural language models.

The NLPL use case, will install, validate, and maintain a selection of these implementations, in an automated and uniform manner, on multiple HPC systems.

ELMo

Embeddings from Language Models (ELMo) use bidirectional LSTM language models to produce contextualized word token representations (Peters et al 2018)

Available implementations

- Reference Tensorflow implementation. Requirements: Python >=3.5, 1.2 < TensorFlow < 1.13 (later versions produce too many deprecation warnings), h5py.

BERT

RoBERTa

ELECTRA