27
$\begingroup$

I've been told that is beneficial to use stratified cross validation especially when response classes are unbalanced. If one purpose of cross-validation is to help account for the randomness of our original training data sample, surely making each fold have the same class distribution would be working against this unless you were sure your original training set had a representative class distribution.

Is my logic flawed?

EDIT I'm interested in whether this method damages the good of CV. I can see why it is necessary if you have a small sample/very unbalanced classes/both to avoid not having a single representative of the minor class in a fold.

The paper Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement puts forward the case for stratification well, but all arguments seem to amount to 'Stratification provides a safeguard and more consistency' but no safeguard would be required given enough data.

Is the answer simply "We use it out of necessity as we rarely have enough data."?

$\endgroup$
17
$\begingroup$

Bootstrapping seeks to simulate the effect of drawing a new sample from the population, and doesn't seek to ensure distinct test sets (residues after N from N sampling with replacement).

RxK-fold Cross-validation ensures K distinct test folds but is then repeated R times for different random partitionings to allow independence assumptions to hold for K-CV, but this is lost with repetition.

Stratified Cross-validation violates the principal that the test labels should never have been looked at before the statistics are calculated, but this is generally thought to be innocuous as the only effect is to balance the folds, but it does lead to loss of diversity (an unwanted loss of variance). It moves even further from the Boostrap idea of constructing a sample similar to what you'd draw naturally from the whole population. Arguably the main reason stratification is important is to address defects in the classification algorithms, as they are too easily biased by over- or under-representation of classes. An algorithm that uses balancing techniques (either by selection or weighting) or optimizes a chance-correct measure (Kappa or preferably Informedness) is less impacted by this, although even such algorithms can't learn or test a class that isn't there.

Forcing each fold to have at least m instances of each class, for some small m, is an alternative to stratification that works for both Bootstrapping and CV. It does have a smoothing bias, making folds tend to be more balanced than they would otherwise be expected to be.

Re ensembles and diversity: If classifiers learned on the training folds are used for fusion not just estimation of generalization error, the increasing rigidity of CV, stratified Bootstrap and stratified CV leads to loss of diversity, and potentially resilience, compared to Bootstrap, forced Bootstrap and forced CV.

$\endgroup$
  • $\begingroup$ Would it be possible please to provide some reference papers on how stratified bootstrap "fails" where force bootstrap "is better"? $\endgroup$ – usεr11852 Jan 30 '18 at 23:44
16
$\begingroup$

Perhaps you can think of it this way. Let's say you have a dataset where there are 100 samples, 90 in class 'A' and 10 in class 'B'. In this very unbalanced design if you do normal randomized groups, you could end up building models on exceedingly few (or EVEN NONE!) from the 'B' class. If you are building a model that is trained on data where there are so few, or even none, of the other class how could you expect it to predict the rarer group effectively? The stratified cross-validation allows for randomization but also makes sure these unbalanced datasets have some of both classes.

To pacify concerns about using stratified CV with more 'balanced' datasets, let's look at an example using R code.

require(mlbench)
require(caret)
require(cvTools)

# using the Sonar dataset (208 samples)
data(Sonar)

# see the distribution of classes are very well balanced
prop.table(table(Sonar$Class))

> prop.table(table(Sonar$Class))

M         R 
0.5336538 0.4663462 

# stratified
# set seed for consistency
# caret::createFolds does stratified folds by default
set.seed(123)
strat <- createFolds(Sonar$Class, k=10)

# non-stratified using cvTools
set.seed(123)
folds <- cvFolds(nrow(Sonar), K=10, type="random")
df <- data.frame(fold = folds$which, index = folds$subsets)
non_strat <- lapply(split(df, df$fold), FUN=function(x) x$index)

# calculate the average class distribution of the folds
strat_dist <- colMeans(do.call("rbind", lapply(strat, FUN = function(x) prop.table(table(Sonar$Class[x])))))
    non_strat_dist <- colMeans(do.call("rbind", lapply(non_strat, FUN = function(x) prop.table(table(Sonar$Class[x])))))
strat_dist
> strat_dist
M         R 
0.5338312 0.4661688 
non_strat_dist
> non_strat_dist
M         R 
0.5328571 0.4671429 

As you can see, in a dataset that is well balanced the folds will have a similar distribution by random chance. Therefore stratified CV is simply an assurance measure in these circumstances. However, to address variance you would need to look at the distributions of each fold. In some circumstances (even starting from 50-50) you could have folds that have splits of 30-70 by random chance (you can run the code above and see this actually happending!). This could lead to a worse performing model because it didn't have enough of one class to accurately predict it thereby increasing overall CV variance. This is obviously more important when you have 'limited' samples where you are more likely to have very extreme differences in distribution.

Now with very large datasets, stratification may not be necessary because the folds will be large enough to still likely contain at least a good proportion of the 'rarer' class. However, there is really no computational loss and no real reason to forgo stratification if your samples are unbalanced no matter how much data you have in my personal opinion.

$\endgroup$
  • $\begingroup$ Yes, this makes total sense. However, this is a very specific case and you're doing it to account for lack of data. If you had 10,000 samples, would you do it? My question is, ideally and given enough data, is it a good idea? $\endgroup$ – James Owers Oct 2 '14 at 17:00
  • $\begingroup$ @kungfujam, it depends on how unbalanced your data is. Even with enormous amounts of data, you could end up with very few of the other class (randomly). There is some research regarding this. Although a little dated, Kohavi reported that stratifcation is generally a better scheme, both in terms of bias and variance, when compared to regular cross-validation. There is no perfect sampling scheme but in unbalanced designs stratification is a good approach. $\endgroup$ – cdeterman Oct 2 '14 at 17:09
  • $\begingroup$ Thank you for this. I'd just found the Kohavi paper. Old but gold. I can see that in general classes are not perfectly balanced and data is limited => stratification is generally better...but given reasonable balance I feel as though it is a violation! $\endgroup$ – James Owers Oct 2 '14 at 17:16
  • $\begingroup$ @kungfujam, what part do you consider a violation? The k-folds are randomized except they specifically require a certain proportion of the different groups. You could think of it as randomly creating your folds from each group and combining them together for one aggregate fold thereby retaining the randomization that you are concerned about. Given a reasonable balance (e.g. 60%-40%) it is likely that your folds will have similar proportions anyway with or without stratification (some variation of course). $\endgroup$ – cdeterman Oct 2 '14 at 17:27
  • $\begingroup$ I feel like it defeats the point. Your original sample is 'random'. As such, I thought that CV was supposed to try and account for this, producing varying resamples, and lead you to produce a more robust model to variance by penalising models that vary as the input data changes. If you restrict your folds to match the proportions of the original sample, I felt like in some sense you're stopping it from doing that. You may well produce a model with a lower bias, but I would reckon it would have a higher variance. $\endgroup$ – James Owers Oct 2 '14 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.