0
$\begingroup$

I am working on a text binary classification problem. I am trying to compare different models such as random forest, linear regression, multinomial naive bayes, sgdc classifier and more using cross_val_score from scikitlearn. They all result in good f1 scores (around 92%). But when I want to predict the class of new observation, they all predict the positive class. Particulary linear regression which seemed to outperform gives the worst class probabilites on new data as opposed to random forest which had lower f1 score. Also, I have an imbalanced dataset (83% positive examples 17% negatives examples) but I used a StratifiedShuffleSplit in cv.

Could someone please enlighten me? Thank you!

$\endgroup$
  • 2
    $\begingroup$ How do you predict? If you simply do >50% probability (and if there is no particularly great predictor), then it might not be surprising. In fact, it might be the best prediction possible by some metrics (e.g. if 60% of records are in one class and there is no way of predicting this from any predictors, then always guessing this class gives you the best accuracy [60%] that you can get). $\endgroup$ – Björn Apr 12 '18 at 16:21
0
$\begingroup$

It is advisable to train a model using a relatively unbiased sample set (especially for small N) or ensure sufficient sample numbers in the smallest class (for large N). You can then test on a realistic sample set (expected ratios).

If you have a very skewed class distribution in training the classifier under-specifies the variation in the low abundance set, and so small deviations that happen randomly can be seized upon by the model and make the classifier appear to perform better than it is. When you then test, these rare differences aren't reproducible and so the performance drops.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.