I was doing multi-class text classification task and I built 2 models, one LSTM model that trains from scratch and other LSTM model with fixed pretrained word embedding.
Fixing the pretrained word embedding gives lower accuracy than training from scratch on both the validation and training data.
What is the reason behind that? Shouldn't the pretrained word give better word representation?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ If you train model on validation dataset it would always have lower validation error… $\endgroup$– Tim ♦Mar 12, 2022 at 11:47
-
$\begingroup$ I didn't train on the validation data. I meant to say that while training both models with keras I observed low accuracy from fixed pretrained word embedding model on the train and validation data $\endgroup$– floydMar 12, 2022 at 15:24
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
I am not sure what you mean by "Fixing the pretrained word embedding", but you are supposed to refine the pre-trained model by training it on your data. This pre-trained model is just supposed to be a good initialization.
However, even if you refine the pre-trained model, the result is often worse than a model trained from scratch.
Pretraining is mainly used if you don't have sufficient data and/or resources to train a large model yourself. Otherwise, the model trained from scratch is often better. It just means that the population the pre-trained model was trained on is too different from your population.
-
$\begingroup$ By "fixing the pretrained word embedding" i meant to do this in code
Embedding(vocab_size, embedding_vector_length, trainable=False)$\endgroup$– floydMar 12, 2022 at 15:20 -
$\begingroup$ "even if you refine the pre-trained model, the result is often worse than a model trained from scratch." But I learnt that there is something called transfer learning and that it makes models that use pretrained weights/layers have higher accuracy than models that learn from scratch. also, models that use pretrained weights/layers should learn faster than training from scratch but that's not what I found in my experiment. $\endgroup$– floydMar 12, 2022 at 15:30
-
1$\begingroup$ @floyd Transfer learning is what I referred to: if you have enough data and enough compute resources and time, fitting from scratch is usually giving you better results than refining transfer-learning models, unless it was pre-trained on data from the same population as your data. Intuitively, it is "easier" to learn from scratch than to first unlearn wrong weights and then relearn the right weights. $\endgroup$– frankMar 12, 2022 at 15:51
-
$\begingroup$ that's the problem in my case. because i am using pretrained word embeddings that were trained on tweets and my data is about tweets. so both of my data sources are from the same population but the pretrained word embedding gives bad accuracy on both training and validation set $\endgroup$– floydMar 13, 2022 at 7:49
-
$\begingroup$ Both being tweets doesn't mean they are from the same population. $\endgroup$– frankMar 13, 2022 at 9:46