Vectors/norlm/norelmo - Revision history

Andreku: /* Evaluation */

2021-03-12T13:16:21Z

‎Evaluation

2021-03-12T13:15:49Z

‎Evaluation

2021-02-16T23:07:55Z

‎Evaluation

2021-02-16T23:07:25Z

‎Evaluation

2021-02-16T23:02:33Z

‎Evaluation

2021-02-16T23:00:35Z

‎Evaluation

2021-02-16T22:56:50Z

‎Evaluation

2021-02-16T22:54:12Z

‎Evaluation

2021-02-16T22:50:00Z

2021-02-14T23:04:55Z

‎NorELMo: Embeddings from Language Models for Norwegian

@@ Line 41: / Line 41: @@
 | Part-of-Speech tagging (accuracy) || 98.0 ''(245)'' || 98.5 (238) || '''98.7''' ''(244)'' || 98.1 ''(8)'' || 98.0 ''(8)''
 |-
-| Fine-grained sentiment analysis (Sentiment Graph F1) || 31.7 ''(444)'' || 34.8 ''(438)'' || 34.8 ''(404)'' || '''34.5''' ''(446)'' || 34.2 ''(434)''
+| Fine-grained sentiment analysis (Sentiment Graph F1) || 31.7 ''(444)'' || '''34.8''' ''(438)'' || '''34.8''' ''(404)'' || 34.5 ''(446)'' || 34.2 ''(434)''
 |-
 | Sentence-level binary sentiment classification (F1 score) || 67.7 ''(37)'' || 77.1 ''(35)'' || '''83.9''' ''(37)'' || 75.0 ''(5)'' || 75.0 ''(5)''

@@ Line 41: / Line 41: @@
 | Part-of-Speech tagging (accuracy) || 98.0 ''(245)'' || 98.5 (238) || '''98.7''' ''(244)'' || 98.1 ''(8)'' || 98.0 ''(8)''
 |-
-| Fine-grained sentiment analysis (Targeted F1) || 34.8 ''(444)'' || 36.9 ''(438)'' || 36.0 ''(404)'' || '''37.9''' ''(446)'' || 37.1 ''(434)''
+| Fine-grained sentiment analysis (Sentiment Graph F1) || 31.7 ''(444)'' || 34.8 ''(438)'' || 34.8 ''(404)'' || '''34.5''' ''(446)'' || 34.2 ''(434)''
 |-
 | Sentence-level binary sentiment classification (F1 score) || 67.7 ''(37)'' || 77.1 ''(35)'' || '''83.9''' ''(37)'' || 75.0 ''(5)'' || 75.0 ''(5)''

@@ Line 37: / Line 37: @@
 {| class="wikitable"
 |-
-! Model/task !! mBERT !! NorBERT !! NB-BERT-Base !! NorELMo30 !! NorELMo100
+! Task/Model !! mBERT !! NorBERT !! NB-BERT-Base !! NorELMo30 !! NorELMo100
 |-
 | Part-of-Speech tagging (accuracy) || 98.0 ''(245)'' || 98.5 (238) || '''98.7''' ''(244)'' || 98.1 ''(8)'' || 98.0 ''(8)''

@@ Line 33: / Line 33: @@
 See the [[Vectors/norlm/norbert#Evaluation|description of the evaluation setup here]]
-In parentheses, the time required to adapt the model to a particular task (fine-tuning or training a classifier over a frozen model) is given in minutes.
+In parentheses, the ''time'' required to adapt the model to a particular task (fine-tuning or training a classifier over a frozen model) is given in ''minutes''.
 {| class="wikitable"
@@ Line 43: / Line 43: @@
 | Fine-grained sentiment analysis (Targeted F1) || 34.8 ''(444)'' || 36.9 ''(438)'' || 36.0 ''(404)'' || '''37.9''' ''(446)'' || 37.1 ''(434)''
 |-
-| Sentence-level binary sentiment classification (F1 score) || 67.7 || 77.1 || '''83.9''' || 75.0 || 75.0
+| Sentence-level binary sentiment classification (F1 score) || 67.7 ''(37)'' || 77.1 ''(35)'' || '''83.9''' ''(37)'' || 75.0 ''(5)'' || 75.0 ''(5)''
 |}

@@ Line 41: / Line 41: @@
 | Part-of-Speech tagging (accuracy) || 98.0 ''(245)'' || 98.5 (238) || '''98.7''' ''(244)'' || 98.1 ''(8)'' || 98.0 ''(8)''
 |-
-| Fine-grained sentiment analysis (Targeted F1) || 34.8 || 36.9 || 36.0 || '''37.9''' || 37.1
+| Fine-grained sentiment analysis (Targeted F1) || 34.8 ''(444)'' || 36.9 ''(438)'' || 36.0 ''(404)'' || '''37.9''' ''(446)'' || 37.1 ''(434)''
 |-
 | Sentence-level binary sentiment classification (F1 score) || 67.7 || 77.1 || '''83.9''' || 75.0 || 75.0

← Older revision		Revision as of 22:56, 16 February 2021
Line 30:		Line 30:

	==Evaluation==		==Evaluation==
		+
		+	See the [[Vectors/norlm/norbert#Evaluation\|description of the evaluation setup here]]

	{\| class="wikitable"		{\| class="wikitable"

@@ Line 35: / Line 35: @@
 ! Model/task !! mBERT !! NorBERT !! NB-BERT-Base !! NorELMo30 !! NorELMo100
 |-
-| Part-of-Speech tagging (accuracy) || 98.0 || 98.5 || '''98.7''' || ||
+| Part-of-Speech tagging (accuracy) || 98.0 || 98.5 || '''98.7''' || 98.1 || 98.0
 |-
-| Fine-grained sentiment analysis (Targeted F1) || 34.8 || '''36.9''' || 36.0 || ||
+| Fine-grained sentiment analysis (Targeted F1) || 34.8 || 36.9 || 36.0 || '''37.9''' || 37.1
 |-
-| Sentence-level binary sentiment classification (F1 score) || 67.7 || 77.1 || '''83.9''' || ||
+| Sentence-level binary sentiment classification (F1 score) || 67.7 || 77.1 || '''83.9''' || 75.0 || 75.0
 |}

@@ Line 28: / Line 28: @@
 ==Training workflow==
 Each models was trained for 3 epochs with batch size 192. We employed a [https://github.com/ltgoslo/simple_elmo_training version of the original ELMo training code updated to work better with the recent TensorFlow versions].  All the hyperparameters were left at their default values, except LSTM dimensionality reduced to 2048 from the default 4096.
 ==Usage==

@@ Line 14: / Line 14: @@
 - ID 211: trained on ''tokenized'' Norwegian Wikipedia Dump of September 2020, about 160 million words ([http://vectors.nlpl.eu/repository/20/210.zip download])
-- ID 217: trained on the [[Vectors/norlm/norbert#Training_Corpus|NorBERT training corpus]] (Wikipedia and Norsk Aviskorpus) with 30 000 words in the target vocabulary ([http://vectors.nlpl.eu/repository/20/217.zip download])
+- ID 217 ('''NorELMo30'''): trained on the [[Vectors/norlm/norbert#Training_Corpus|NorBERT training corpus]] (Wikipedia and Norsk Aviskorpus) with 30 000 words in the target vocabulary ([http://vectors.nlpl.eu/repository/20/217.zip download])
-- ID 218: trained on the [[Vectors/norlm/norbert#Training_Corpus|NorBERT training corpus]] (Wikipedia and Norsk Aviskorpus) with 100 000 words in the target vocabulary ([http://vectors.nlpl.eu/repository/20/217.zip download])
+- ID 218 ('''NorELMo100'''): trained on the [[Vectors/norlm/norbert#Training_Corpus|NorBERT training corpus]] (Wikipedia and Norsk Aviskorpus) with 100 000 words in the target vocabulary ([http://vectors.nlpl.eu/repository/20/218.zip download])
 ==Preprocessing==