2
$\begingroup$

I want to investigate why am I seeing the below described phenomena. I welcome all the logical explanation which might hint towards what is happening.

So I have a dataset which contains two classes:

  • 0
  • 1

So its a basic binary classification. I have used algorithms such as:

  • Logistic Regression
  • Decision Tree
  • Random Forest
  • Gradient Boosting

I did a train-test split just to see if my model is overfitting or not. The train data contains class 0 which is 85% of the data and class 1 which is 15%. The train accuracy I achieved is about 90% and the test accuracy is about 89%. But when I am predicting it on real-test set, I am getting weird proposition of classes. I was expecting class 0 and class 1 proportion to be around what it was in train set that is 85% to 15% or nearby. But after predictive modelling I am getting the ratio as 60%:40% for class 0 and class 1.

Now I know for a fact that the proportion cannot be that much as the data which I am classifying cannot have class 1 so high. I have tried thinking of what could be the reason but cannot come up with anything. Since its a test set I cannot verify it, but only assume that what I am getting is right. I am confused whether to use this analysis or not. If anyone can please shed some light on what am I doing wrong, would be great. Please dont criticize my post as I tried explaining it to my best ability.

EDIT:

Also, one point to add here is, different algorithm such as logistic, random forest etc I am getting different proportions. For ex Decision tree is giving me 30 % Class 1, Logistic is inflating it to 66 %, Random is giving 40%.

EDIT 2:

I tired something new. So I had 5 independent variables which i was using to predict. I tried removing 1 of them and then the class proportion went from 60% to 10%, something what it should be. I did random removals and every time the proportion changes where as the accuracy for train and validation remains the same. Why is this happening ?

$\endgroup$

closed as unclear what you're asking by Michael Chernick, mdewey, mkt, Peter Flom Sep 15 '18 at 12:01

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Welcome to the site :) What do you mean by real-test set vs test set? real-test set being the one where you do not know the labels? It may be the case that the class distribution from that real-test set is not the same as the one in your training and validation sets? $\endgroup$ – Tom Sep 14 '18 at 13:51
  • $\begingroup$ Thank you Tom. Yes real test set is the one which I don't know the labels for. I am trying to see the distribution of data using Visualization just to see if the underlying data varies or not. That may be a cause of this.. But the problem which I am facing is that using different algorithm such as logistic, random forest etc I am getting different proportions. For ex Decision tree is giving me 30 % Class 1, Logistic is inflating it to 66 %, Random is giving 40%. So I think the underlying data may not have to do anything with it. Is there anything I am missing building the model here? $\endgroup$ – Kshitij Yadav Sep 14 '18 at 13:56
  • $\begingroup$ What do you mean by "Since its a test set I cannot verify it"? You should be able to verify on the test set. $\endgroup$ – Peter Flom Sep 15 '18 at 12:00
0
$\begingroup$

First thing first, I don't recommend using only accuracy as your success metric for imbalanced datasets. Support it with metrics like precision, recall, etc. For example, a 60-40 % separation in your test set might correspond to 75 % accuracy (i.e. know 60 of 85 as Class 1, and the other 40 would include the remaining 15 Class 2 samples)

For your case, there might be the following cases:

1) You didn't train & validate your algorithms very well.

2) Your real test set class distribution is not the same as your training/test set distributions as @Tom suggests.

3) Most ML methods assumes your training & test sets come from the same underlying population. When this is not true, i.e. your real test set is coming from another distribution/population; the classical ways of learning fail. This is not a trivial problem, and there are theses for dealing such issues.

$\endgroup$
  • $\begingroup$ OK so i tired something new. So I had 5 independent variables which i was using to predict. I tried removing 1 of them and then the class proportion went from 60% to 10%, something what it should be. I did random removals and every time the proportion changes where as the accuracy for train and validation remains the same. Why is this happening ? $\endgroup$ – Kshitij Yadav Sep 14 '18 at 14:37
  • $\begingroup$ If 10 % is your class 0, it's not as it should've been, right? What are your classifiers' proportions for training & validation? $\endgroup$ – gunes Sep 14 '18 at 14:48
  • $\begingroup$ 10% for training and 9.8% for test, but I want to know why different algo gives different proportion and also why removing some variables gives me different proportion. Any idea ? $\endgroup$ – Kshitij Yadav Sep 14 '18 at 15:54
  • $\begingroup$ First of all, your different algorithms is apparently learning differently. Have you done hyper parameter tuning for all of them with a validation set? Maybe some of them are not trained very well. $\endgroup$ – gunes Sep 14 '18 at 15:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.