2
$\begingroup$

I learned that if we fine-tune a task based on a pre-trained model and the vocab of the new task is relatively small compared to the original pre-trained model we usually fix the embedding layer.

enter image description here

But the issue I encountered is that we have some out of vocabulary(OOVs) words in the new task?

For example, we are doing transfer learning and the original vocabulary is of size 200000, and the new dataset for fine-tuning has a vocabulary of only 3010 but 10 of which are new words for the original 200000 vocab?

Some OOVs are very similar to the other words in the original vocabulary and I just replaced the original ones with the new ones but some are very different from those in the original vocab.

$\endgroup$

2 Answers 2

1
$\begingroup$

you can now do this. Keras nightly version has added a new util keras.utils.warmstart_embedding_matrix. Using this you can continuously train your model with changing vocabulary. https://www.tensorflow.org/api_docs/python/tf/keras/utils/warmstart_embedding_matrix

$\endgroup$
0
$\begingroup$

Just add a new embedding layer for the new words and freeze the pretrained layer and make the new layer trainable. Here is a related answer: Is it possible to freeze only certain embedding weights in the embedding layer in PyTorch?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.