I want to investigate why am I seeing the below described phenomena. I welcome all the logical explanation which might hint towards what is happening.
So I have a dataset which contains two classes:
So its a basic binary classification. I have used algorithms such as:
- Logistic Regression
- Decision Tree
- Random Forest
- Gradient Boosting
I did a train-test split just to see if my model is overfitting or not. The train data contains class 0 which is 85% of the data and class 1 which is 15%. The train accuracy I achieved is about 90% and the test accuracy is about 89%. But when I am predicting it on real-test set, I am getting weird proposition of classes. I was expecting class 0 and class 1 proportion to be around what it was in train set that is 85% to 15% or nearby. But after predictive modelling I am getting the ratio as 60%:40% for class 0 and class 1.
Now I know for a fact that the proportion cannot be that much as the data which I am classifying cannot have class 1 so high. I have tried thinking of what could be the reason but cannot come up with anything. Since its a test set I cannot verify it, but only assume that what I am getting is right. I am confused whether to use this analysis or not. If anyone can please shed some light on what am I doing wrong, would be great. Please dont criticize my post as I tried explaining it to my best ability.
Also, one point to add here is, different algorithm such as logistic, random forest etc I am getting different proportions. For ex Decision tree is giving me 30 % Class 1, Logistic is inflating it to 66 %, Random is giving 40%.
I tired something new. So I had 5 independent variables which i was using to predict. I tried removing 1 of them and then the class proportion went from 60% to 10%, something what it should be. I did random removals and every time the proportion changes where as the accuracy for train and validation remains the same. Why is this happening ?