Why use stratified cross validation? Why does this not damage variance related benefit?

Question

I've been told that is beneficial to use stratified cross validation especially when response classes are unbalanced. If one purpose of cross-validation is to help account for the randomness of our original training data sample, surely making each fold have the same class distribution would be working against this unless you were sure your original training set had a representative class distribution.

Is my logic flawed?

EDIT I'm interested in whether this method damages the good of CV. I can see why it is necessary if you have a small sample/very unbalanced classes/both to avoid not having a single representative of the minor class in a fold.

The paper Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement puts forward the case for stratification well, but all arguments seem to amount to 'Stratification provides a safeguard and more consistency' but no safeguard would be required given enough data.

Is the answer simply "We use it out of necessity as we rarely have enough data."?

Answer 1

Bootstrapping seeks to simulate the effect of drawing a new sample from the population, and doesn't seek to ensure distinct test sets (residues after N from N sampling with replacement).

RxK-fold Cross-validation ensures K distinct test folds but is then repeated R times for different random partitionings to allow independence assumptions to hold for K-CV, but this is lost with repetition.

Stratified Cross-validation violates the principal that the test labels should never have been looked at before the statistics are calculated, but this is generally thought to be innocuous as the only effect is to balance the folds, but it does lead to loss of diversity (an unwanted loss of variance). It moves even further from the Boostrap idea of constructing a sample similar to what you'd draw naturally from the whole population. Arguably the main reason stratification is important is to address defects in the classification algorithms, as they are too easily biased by over- or under-representation of classes. An algorithm that uses balancing techniques (either by selection or weighting) or optimizes a chance-correct measure (Kappa or preferably Informedness) is less impacted by this, although even such algorithms can't learn or test a class that isn't there.

Forcing each fold to have at least m instances of each class, for some small m, is an alternative to stratification that works for both Bootstrapping and CV. It does have a smoothing bias, making folds tend to be more balanced than they would otherwise be expected to be.

Re ensembles and diversity: If classifiers learned on the training folds are used for fusion not just estimation of generalization error, the increasing rigidity of CV, stratified Bootstrap and stratified CV leads to loss of diversity, and potentially resilience, compared to Bootstrap, forced Bootstrap and forced CV.

Answer 2

Perhaps you can think of it this way. Let's say you have a dataset where there are 100 samples, 90 in class 'A' and 10 in class 'B'. In this very unbalanced design if you do normal randomized groups, you could end up building models on exceedingly few (or EVEN NONE!) from the 'B' class. If you are building a model that is trained on data where there are so few, or even none, of the other class how could you expect it to predict the rarer group effectively? The stratified cross-validation allows for randomization but also makes sure these unbalanced datasets have some of both classes.

To pacify concerns about using stratified CV with more 'balanced' datasets, let's look at an example using R code.

require(mlbench)
require(caret)
require(cvTools)

# using the Sonar dataset (208 samples)
data(Sonar)

# see the distribution of classes are very well balanced
prop.table(table(Sonar$Class))

> prop.table(table(Sonar$Class))

M         R 
0.5336538 0.4663462 

# stratified
# set seed for consistency
# caret::createFolds does stratified folds by default
set.seed(123)
strat <- createFolds(Sonar$Class, k=10)

# non-stratified using cvTools
set.seed(123)
folds <- cvFolds(nrow(Sonar), K=10, type="random")
df <- data.frame(fold = folds$which, index = folds$subsets)
non_strat <- lapply(split(df, df$fold), FUN=function(x) x$index)

# calculate the average class distribution of the folds
strat_dist <- colMeans(do.call("rbind", lapply(strat, FUN = function(x) prop.table(table(Sonar$Class[x])))))
    non_strat_dist <- colMeans(do.call("rbind", lapply(non_strat, FUN = function(x) prop.table(table(Sonar$Class[x])))))
strat_dist
> strat_dist
M         R 
0.5338312 0.4661688 
non_strat_dist
> non_strat_dist
M         R 
0.5328571 0.4671429 

As you can see, in a dataset that is well balanced the folds will have a similar distribution by random chance. Therefore stratified CV is simply an assurance measure in these circumstances. However, to address variance you would need to look at the distributions of each fold. In some circumstances (even starting from 50-50) you could have folds that have splits of 30-70 by random chance (you can run the code above and see this actually happending!). This could lead to a worse performing model because it didn't have enough of one class to accurately predict it thereby increasing overall CV variance. This is obviously more important when you have 'limited' samples where you are more likely to have very extreme differences in distribution.

Now with very large datasets, stratification may not be necessary because the folds will be large enough to still likely contain at least a good proportion of the 'rarer' class. However, there is really no computational loss and no real reason to forgo stratification if your samples are unbalanced no matter how much data you have in my personal opinion.