Difference between revisions of "Eosc/pretraining/nvidia"

From Nordic Language Processing Laboratory
Jump to: navigation, search
(Created page with "= Background = This page provides a recipe to large-scale pre-training of a BERT neural language model, using the [https://github.com/NVIDIA/DeepLearningExamples/tree/master/...")
 
(Background)
Line 2: Line 2:
  
 
This page provides a recipe to large-scale pre-training of a
 
This page provides a recipe to large-scale pre-training of a
BERT neural language model, using the
+
BERT neural language model, using the high-efficiency
 
[https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT NVIDIA BERT] implementation
 
[https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT NVIDIA BERT] implementation
 
(which is based on TensorFlow, in contrast to the
 
(which is based on TensorFlow, in contrast to the

Revision as of 10:11, 7 November 2020

Background

This page provides a recipe to large-scale pre-training of a BERT neural language model, using the high-efficiency NVIDIA BERT implementation (which is based on TensorFlow, in contrast to the NVIDIA Megatron code).

Software Installation

Data Preparation

Training Example