3
$\begingroup$

In the case of binary classification, stratified cross-validation only ensures that each fold contains roughly the same proportions of the two types of class labels.

When does it make sense to also ensure that the feature distribution is maintained?

(I would expect most algorithms to be biased by not only the class distribution)

$\endgroup$
  • 1
    $\begingroup$ In general you assume that your training and test set come from the same distribution, which makes learning possible. That's one of the more important assumptions in supervised ML. However, it may not be the case for different reason: at test time, you might use a different data collecting device which produces different type of noise for instance. It is also not the case in zero-shot learning applications. $\endgroup$ – Tom Sep 6 '18 at 17:06
3
+50
$\begingroup$

Imbalances in features do have an impact on some ML algorithms (typically, the more your algorithm looks like an ensemble method -e.g. random forests-, the less it will be impacted by feature imbalances).

I wrote a blog post a short while ago showing an example of how you could address feature imbalances using a reweighting method (I used it originally to improve the results of a soccer prediction algorithm for which feature imbalances had a noticeable impact on the predicted probabilities of one of the classes).

Cross-validation will work the same: if the distribution of classes and features have a high correlation, then the cross-validation estimate will be affected. There are a lot of possible solutions. A quick one would be using reweighting observations just like in the blog post I mentioned (could be very efficient for example if you know exactly which features have a high impact on the performance of the algorithm). Other solutions come from subsampling and bootstrap literature (see for example Kim, 2009 or Bengio & Grandvalet, 2005).

References

Bengio, Y., & Grandvalet, Y. (2005). Bias in estimating the variance of k-fold cross-validation. Statistical modeling and analysis for complex data problems, 75–95.

Kim, J.-H. (2009). Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational Statistics and Data Analysis, 53(11), 3735–3745. doi:10.1016/j.csda.2009.04.009

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.