Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In the case of binary classification, stratified cross-validation only ensures that each fold contains roughly the same proportions of the two types of class labels.
When does it make sense to also ensure that the feature distribution is maintained?
(I would expect most algorithms to be biased by not only the class distribution)
Imbalances in features do have an impact on some ML algorithms (typically, the more your algorithm looks like an ensemble method -e.g. random forests-, the less it will be impacted by feature imbalances).
I wrote a blog post a short while ago showing an example of how you could address feature imbalances using a reweighting method (I used it originally to improve the results of a soccer prediction algorithm for which feature imbalances had a noticeable impact on the predicted probabilities of one of the classes).
Cross-validation will work the same: if the distribution of classes and features have a high correlation, then the cross-validation estimate will be affected. There are a lot of possible solutions. A quick one would be using reweighting observations just like in the blog post I mentioned (could be very efficient for example if you know exactly which features have a high impact on the performance of the algorithm). Other solutions come from subsampling and bootstrap literature (see for example Kim, 2009 or Bengio & Grandvalet, 2005).
Bengio, Y., & Grandvalet, Y. (2005). Bias in estimating the variance of k-fold cross-validation. Statistical modeling and analysis for complex data problems, 75–95.
Kim, J.-H. (2009). Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational Statistics and Data Analysis, 53(11), 3735–3745. doi:10.1016/j.csda.2009.04.009
