6
$\begingroup$

I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced while building the model and making predictions due to missingness in covariates. I observed that the "complete" training set had half the $Y=1$ cases compared to the "complete" test set.

The AUC for the training data was about 0.70 and the AUC of the test data was about 0.85.

How should I explain that? I thought the training data would always show higher AUC than the test data because we used training data to build our model.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Did you split the data randomly into train / test? $\endgroup$
    – user2974951
    Commented Dec 4, 2018 at 10:35
  • $\begingroup$ Yes, I did it randomly. But there were missing covariates in the data so while building the model it only used the "complete" training data or observations which had values in all covariates. Similarly, for prediction, it would only give me a prediction for "complete" test data. Those "complete" data didn't have 25% $Y=1$ each. Sorry that it was not clear from my question. $\endgroup$
    – Blain Waan
    Commented Dec 4, 2018 at 10:40
  • $\begingroup$ Try 10 different test-train splits. How big is your data? Could just be that you got lucky. A good sanity check is to test that the mean and standard deviation is equal between test and training sets $\endgroup$
    – Jon Nordby
    Commented Dec 5, 2018 at 2:34
  • $\begingroup$ How did the data 'become unbalanced' during training? If you are removing samples to deal with missing values, you should probably do that before splitting into test/train splits. If you want a model that has validity for such samples then either impute the missing values, or remove the features which has missing values. $\endgroup$
    – Jon Nordby
    Commented Dec 5, 2018 at 2:38

1 Answer 1

Reset to default
2
$\begingroup$

This can be easily attributed to random variation. While, indeed the in-sample performance is expected better than the out-of-sample performance (i.e. our training error be less than our test error), that is not a necessity; as the AUC value calculated here is a statistic, a function of our present sample, it is subject to sampling variability. It would be reasonable to use multiple training/test splits (i.e. bootstrap the sample at hand) so we are able to quantify the variability of that statistic. Repeated cross-validation and/or bootstrapping are standard approaches to estimate the sampling distribution of a statistic of interest. There a very informative thread in CV on: Hold-out validation vs. cross-validation that I think will help clarify things even further.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks. The discussion on hold-out and cross-validation looks useful. $\endgroup$
    – Blain Waan
    Commented Feb 22, 2019 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.