Eosc/NorBERT3 corpus - Revision history

Andreku: /* Sampling experiment */

2022-10-24T18:50:42Z

‎Sampling experiment

2022-10-24T18:49:32Z

‎To Decide

2022-10-22T23:57:27Z

‎Workflow

2022-10-21T15:50:02Z

‎To Decide

2022-10-21T14:37:48Z

‎To Decide

2022-10-18T21:41:35Z

‎To Decide

2022-10-18T20:14:18Z

Cleaning

2022-10-12T14:27:27Z

2022-10-12T14:25:36Z

2022-10-12T14:20:42Z

@@ Line 8: / Line 8: @@
 We plan to create two versions of the training corpus:
 * baseline (as is)
-* Wikipedia+NCC+NAK multiplied by two to match the C4 size (oversampling quality data)
+* Wikipedia+NCC+NAK+NBDigital multiplied by two to match the C4 size (oversampling quality data)
 == Vocabulary ==

@@ Line 14: / Line 14: @@
 == To Decide ==
-The size of NBDigital is 662M tokens. Should we use it? It probably overlaps a lot with NCC.
+Q: The size of NBDigital is 662M tokens. Should we use it? It probably overlaps a lot with NCC.
-How should we split training corpora: one sentence per line, one paragraph per line, one document per line?
+A: No, it isn't. Only 60 paragraphs out of total 18M in NCC are duplicates of paragraphs in NBDigital. Thus, we definitely should use it.
 A: BERT assumes that there is one sentence per line.

@@ Line 1: / Line 1: @@
 == Workflow ==
 * [https://arxiv.org/abs/2112.11446 Cleaning]
-* [https://github.com/ChenghaoMou/text-dedup/tree/main/text_dedup Deduplication] ([https://pypi.org/project/mmh3/ MurMurHash])
-* [https://github.com/ekzhu/datasketch ...or another package]
+There are [https://github.com/ekzhu/datasketch other de-duplication packages]
 == Sampling experiment ==

← Older revision		Revision as of 15:50, 21 October 2022
Line 16:		Line 16:

	How should we split training corpora: one sentence per line, one paragraph per line, one document per line?		How should we split training corpora: one sentence per line, one paragraph per line, one document per line?
		+
		+	A: BERT assumes that there is one sentence per line.

← Older revision		Revision as of 14:37, 21 October 2022
Line 14:		Line 14:
	== To Decide ==		== To Decide ==
	The size of NBDigital is 662M tokens. Should we use it? It probably overlaps a lot with NCC.		The size of NBDigital is 662M tokens. Should we use it? It probably overlaps a lot with NCC.
		+
		+	How should we split training corpora: one sentence per line, one paragraph per line, one document per line?

← Older revision		Revision as of 21:41, 18 October 2022
Line 13:		Line 13:

	== To Decide ==		== To Decide ==
−	~~What is the~~ size of NBDigital ~~and should~~ we use it? It probably overlaps a lot with NCC.	+	The size of NBDigital is 662M tokens. Should we use it? It probably overlaps a lot with NCC.

← Older revision		Revision as of 14:27, 12 October 2022
Line 5:		Line 5:
	Todo: what is the size of NBDigital and should we use it?		Todo: what is the size of NBDigital and should we use it?

−	Todo: vocabulary size?	+	Todo: vocabulary size? Start with 50K

← Older revision		Revision as of 14:25, 12 October 2022
Line 4:		Line 4:

	Todo: what is the size of NBDigital and should we use it?		Todo: what is the size of NBDigital and should we use it?
		+
		+	Todo: vocabulary size?

← Older revision		Revision as of 14:20, 12 October 2022
Line 2:		Line 2:
	* Deduplication https://github.com/ChenghaoMou/text-dedup/tree/main/text_dedup https://github.com/ekzhu/datasketch		* Deduplication https://github.com/ChenghaoMou/text-dedup/tree/main/text_dedup https://github.com/ekzhu/datasketch
	* Two versions: baseline and wikipedia+NCC+NAK multiplied by two to match the C4 size		* Two versions: baseline and wikipedia+NCC+NAK multiplied by two to match the C4 size
		+
		+	Todo: what is the size of NBDigital and should we use it?