I am training a tagger to predict whether or not a "word" is a proper noun or not. To do this I take in a list of "words" and their tags for part of speech. I then change all tags that aren't the tag for proper noun to a singular tag. From there I run a method to extract certain features from this data. I train a logistic regression model using train test split.

My model was performing with about a 95.5% accuracy. Then I added a second data set (formatted the same) prior to any manipulation of the input data - essentially just elongating the data I was already using. This dropped my accuracy about 2 percentage points, the complete opposite of what I was expecting. No other code was changed in between the tests.

Are there any common reasons for this sort of issue? I never would have thought adding data (a significant amount) could result in a lower accuracy. Additionally, is there a way to fix this problem? Thank you to anyone who helps!

  • $\begingroup$ When you say "accuracy", do you mean in sample or out of sample accuracy? $\endgroup$ – Cliff AB Apr 3 at 16:06
  • $\begingroup$ @CliffAB In sample accuracy $\endgroup$ – Anthony Girard Apr 4 at 0:17
  • $\begingroup$ In sample accuracy can be subject to overfitting, which gets harder when you have more data $\endgroup$ – Cliff AB Apr 4 at 1:06

There are quite a few reasons this could happen. Probably the two most important to be aware of is:

1) Your accuracy didn't really drop.

Every measure of model performance is subject to randomness. I.e., it's sensitive to the particular data you used to train and test your model. Its possible that you just randomly ended up with a "hard" test set or an "easy" training set. It's best to get multiple measures of model performance, and then average them, this gives you a much more stable look at your situation. The main tool for doing this is bootstrapping.

2) Your second chunk of data is not sampled from the same population as the first.

It's possible that your second set of data is just describing a slightly different phenomena than the first. Like, if you first set was from twitter and your second set was from facebook. Or, more subtly, your first set was tweets from random days, but your second were tweets sent on christmas (or any other holiday).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.