1
$\begingroup$

so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.

However, when I remove a specific similar sentence from the training data and add it to my test data, the classifier fails to predict the class of that sentence. For example, sentiment analysis task

"I love the movie more specifically the acting was great"

I removed from training all sentences containing the words " more specifically" and surprisingly in the test set they were all misclassified, so the precision decreased by a huge amount.

Any ideas on how can I further fine-tune/improve my model to work better on unseen text in training to avoid the problem I described above? (of course without feeding the model on sentences containing the words "more specifically")

Note: I observed the same performance regardless of the language model in use (BERT, RoBERTa etc).

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

Interesting. There are reasons why this sentence is hard to process, so some training examples may be needed.

Grammatically, it is two sentences not one - both "loved" and "was" are root words, not dependent on any other. There is no punctuation to indicate this. If your model was pretrained on well formed sentences, then it is not surprising that it is struggling.

Secondly, the use of the word "specifically" is an unusual one. It indicates that the following sentence is more specific than the preceding one, but neither are stated as a complement. i.e. it has a similar meaning to, but is stated differently to, "I love the movie, and to be specific I love the acting."

This use of specific is similar to the German use of "Hoffentlich" / the US English "Hopefully", where the listener is suppose to understand that the hopeful person is the (unnamed) speaker. This is not standard usage in British English.

One approach would be to train with similar sentence using adverbs other than "specifically", e.g.

  • I love this movie more generally I love everything with Tom Cruise in it.
  • I did not like this move more precisely I hated the ending
  • I hate that book more emphatically I loathe it
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks Chris for your answer! Yes what I did is mainly data augmentation, using back-translation and also NLP augmenting tools like textAttack. Yet the problem that I don't want to use the sentences containing more specifically in my training data (or even its derivatives from back-translation and text attack, but supposedly I don't know the label, so I can't add it in my training data). $\endgroup$
    – IS92
    Commented Jan 4, 2022 at 15:54
  • $\begingroup$ Yet, what I did is multi-task fine-tuning the LM by training on a very similar task (so I did not train on the original training set at all), it achieved an overall F-measure of approximately ~45%, however, sentences containing more specifically were misclassified 170 times only (compared when training on the original training data sentences containing more specifically were misclassified 750 times!! $\endgroup$
    – IS92
    Commented Jan 4, 2022 at 15:55
  • $\begingroup$ But I can't directly use the output of the multi-task fine-tuning because overall results are bad, I was thinking of some kind of ensemble but no sure how to design it, I would like to hear your thoughts on this :). $\endgroup$
    – IS92
    Commented Jan 4, 2022 at 15:55
2
$\begingroup$

So now you have one model that works OK on sentences that contain "more specifically", and one that works on other sentences.

Then the right predictor would seem to be:

model1(sentence) if 'more specifically' in sentence else model2(sentence)

If your goal is simply to make good predictions, then that would seem to ge good enough. Maybe the condition can be generalised a little to regexp 'more \w+ly ', or maybe not. I suspect that the non-grammatical nature of the sentence may be part of the issue.

Alternatively, if you want to know you have made a single neural net model, you could take the second model, then retrain it with the movies test set and a low learning rate. That might improve it, without making it forget its success on these sentence. Or not!

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes thats right: model1(sentence) if 'more specifically' in sentence else model2(sentence), yet I can't do that because here more specifically is the only APPARENT problem now, but maybe in the future I also get "in particularly" in my test set and it is also misclassified, so I was hoping for something more general. I have been at this issue for almost a month now actually, it is not easy and I am quite surprised that pretrained language models behave like that. $\endgroup$
    – IS92
    Commented Jan 7, 2022 at 10:07
  • $\begingroup$ The 2nd part of your answer is really interesting! I will give it a try and let you know how it worked out (I hope I am not bothering you). $\endgroup$
    – IS92
    Commented Jan 7, 2022 at 10:08
  • $\begingroup$ So I assigned a lower learning rate to the unrelated dataset (dataset 1) and a higher learning rate to the original training set. (dataset 2), and I fine-tuned pretrained BERT on dataset 1 first then 2nd stage fine-tuning on dataset 2. Yet, no much improvement the performance is fluctuating between 81%~84% (which is the same performance when I use data augmentation like back translation and so on ) Not sure if it will differ or not, but maybe I could first fine-tune on dataset 2 ( the original dataset with higher learning rate) then dataset 1 (the related dataset ower learning rate). $\endgroup$
    – IS92
    Commented Jan 8, 2022 at 10:05
1
$\begingroup$

I'm not clear on the problem.

You say "The results are good almost 98% F-measure." Is that on a test set, independent of the training data? If so then I am confused by "when I remove a specific similar sentence from the training data and add it to my test data, the classifier fails to predict the class of that sentence".

If you mean that the achieves 98% F-measure on its training data, but not on test data that it has not been trained on, then that is a clear case of overfitting.

The fix for overfitting is more data - or if you can't get more data, abit of noise or another form of regularization.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes the 98% F-measure is on the test set, when I shuffle all the data and do random split to create a training set and a test set. The problem comes when I deliberately leave sentences containing more specifically" out of training set, which results in all sentences containing more specifically in test set being misclassified, so the F-measure drops to 80% in that case. $\endgroup$
    – IS92
    Commented Dec 23, 2021 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.