I am working on a text binary classification problem. I am trying to compare different models such as random forest, linear regression, multinomial naive bayes, sgdc classifier and more using cross_val_score from scikitlearn. They all result in good f1 scores (around 92%). But when I want to predict the class of new observation, they all predict the positive class. Particulary linear regression which seemed to outperform gives the worst class probabilites on new data as opposed to random forest which had lower f1 score. Also, I have an imbalanced dataset (83% positive examples 17% negatives examples) but I used a StratifiedShuffleSplit in cv.
Could someone please enlighten me? Thank you!