Lumi/pilot - Revision history

Andreku: /* Model Architectures */

2022-10-12T14:13:49Z

‎Model Architectures

2022-08-10T10:34:32Z

‎Software Support

2022-08-10T10:32:40Z

‎Model Architectures

2022-08-08T15:45:23Z

‎Software Support

2022-07-01T13:47:48Z

‎Data: Norwegian

2022-07-01T13:45:06Z

‎Software Support

2022-07-01T13:40:55Z

‎LUMI-G Pilot

2022-06-01T12:17:57Z

‎Model Architectures

2021-03-25T23:54:38Z

‎Data: Norwegian

2021-03-25T23:53:58Z

‎Model Architectures

@@ Line 16: / Line 16: @@
 Prioritized:
-* [[Eosc/pretraining#BERT|NorBERT 3]] on the concatenation of NorBERT1/NorBERT2 corpora (''base'' and ''large'' versions)
+* [[Eosc/pretraining#BERT|NorBERT 3]] on the [[Eosc/NorBERT3_corpus|concatenation of NorBERT1/NorBERT2 corpora]] (''base'' and ''large'' versions)
 * Separate BERT-base models for Bokmål and Nynorsk
 * [https://arxiv.org/pdf/1910.10683.pdf T5] on the NorBERT3 corpus: at least the unsupervised denoising objective stage

@@ Line 36: / Line 36: @@
 Multi-GPU and multi-node training must be possible. In the NVIDIA world, [https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/overview.html NCCL] and [https://github.com/horovod/horovod Horovod] are used for this.
-In the AMD world? No idea.
+In the AMD world? [https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/ MIOpen & RCCL] ?
 = Data: Norwegian =

@@ Line 14: / Line 14: @@
 = Model Architectures =
 * [https://arxiv.org/pdf/1910.10683.pdf T5]
 * Ablations with BERT
 * [[Eosc/pretraining#ELECTRA|ELECTRA]]
-* [[Eosc/pretraining#BERT|BERT]] (separate Bokmål and Nynorsk models)
+*  (separate Bokmål and Nynorsk models)
 * [[Eosc/pretraining#RoBERTa|RoBERTa]]
-* [https://openai.com/blog/gpt-2-1-5b-release/ GPT]
+* Large language models with linguistically motivated inductive biases (linked to David Samuel PhD topic); one example is Google's [https://www.aclweb.org/anthology/2020.emnlp-main.19/ ETC].
 = Software Support =

@@ Line 25: / Line 25: @@
 See the links above for particular model's requirements.
-In general, we rely on Python (>=3.8) and its [https://www.scipy.org/ SciPy] stack.
+In general, we rely on Python (>=3.9) and its [https://www.scipy.org/ SciPy] stack.
 We definitely will require fully functional GPU-enabled installations of PyTorch (1.11) and TensorFlow (preferably, both 1.15.5 and 2.8.2).

@@ Line 34: / Line 34: @@
 = Data: Norwegian =
-* Collaboration with the National Library ([https://github.com/NBAiLab/notram/blob/master/guides/corpus_description.md Colossal Norwegian Corpus])?
+* Collaboration with the National Library ([https://github.com/NBAiLab/notram/blob/master/guides/corpus_description.md Colossal Norwegian Corpus]): we now have the public part of it (''/cluster/projects/nn9851k/corpora/NCC'' on Saga)
-* Extracting the Norwegian part from the [https://github.com/allenai/allennlp/discussions/5056 C4 dataset]?
+* Extracting the Norwegian part from the [https://github.com/allenai/allennlp/discussions/5056 C4 dataset]: ''/cluster/projects/nn9851k/corpora/c4'' on Saga
-* Additional news collections (Lilja?)
+* Additional news collections from MediaFutures SFI (Lilja?)

@@ Line 1: / Line 1: @@
-= LUMI-G Pilot =
+= Very Large Language Models in the Nordics (VLLMN) =
 [[File:norbert.png|thumb|right|150px]]
-In late 2021, the shared
+In the summer of 2022, the shared
 [https://www.lumi-supercomputer.eu/ LUMI supercomputer]
 will (likely) open for trial usage of its vast gpu partition.

← Older revision		Revision as of 23:54, 25 March 2021
Line 35:		Line 35:
	* Collaboration with the National Library ([https://github.com/NBAiLab/notram/blob/master/guides/corpus_description.md Colossal Norwegian Corpus])?		* Collaboration with the National Library ([https://github.com/NBAiLab/notram/blob/master/guides/corpus_description.md Colossal Norwegian Corpus])?
	* Extracting the Norwegian part from the [https://github.com/allenai/allennlp/discussions/5056 C4 dataset]?		* Extracting the Norwegian part from the [https://github.com/allenai/allennlp/discussions/5056 C4 dataset]?
		+	* Additional news collections (Lilja?)