I had a discussion with my colleague about Kfold and StratifiedKfold validation in scikit learn.

My understanding of these two ways of validation is that at the beginning of the validation process, the data set is randomly divided in k-folds. k-1 of those folds end up in the train dataset. 1 of those folds ends up in the validation data set. Validations are performed k times. With each repetition, what is in the training or validation dataset is shifted.

My colleague argued that it would be better to decide what is and is not in the train dataset or validation dataset anew. Edit: Scikit-learn also has selectors for this sort of validation named "ShuffleSplit" and "StratifiedSHuffling".

I had no mathematical intuition for why one way is superior to another way.

The benefit of only doing the initial split is that what ends up in the validation dataset is non-redundant. But is that really a benefit?

So my question is this:

Is he right? What is the benefit (intuitively or mathematically) to the actual implementation of Kfold or StratifiedKfold?

Edit: Can someone provide me with an intuition for what to pick when?

  • $\begingroup$ The benefit of only 1 split is that all samples are used both for training ($k-1$) times and exactly once for validation. Whereas the advantage of random sub-sampling is that the ratio of training to test size is independent of number of folds. $\endgroup$ – Łukasz Grad Mar 29 '17 at 18:49
  • $\begingroup$ @Lukasz Grad: Which is better for what cases? Is there a sweet-spot for doing only 1 split at the beginning? $\endgroup$ – Thornhale Mar 29 '17 at 19:00
  • $\begingroup$ Another option for smaller datasets is iterated K-fold validation with shuffling. Just repeat K-fold multiple times, shuffling only at the beginning of each. So (# iterations)*K trained models in total. $\endgroup$ – Austin Jul 7 '18 at 1:07

The benefit of determining the contents of each fold at the beginning rather than re-sampling is that you avoid bias by possibly selecting a single observation for more than one training or testing set. This is only a benefit if you care that no records are over-represented in your validation.

If the argument is that new folds should be generated without replacement after each fold has been tested, it can be shown that the likelihood of a single observation to appear in each fold is unaffected and they are therefore equivalent approaches.

Stratified methods most commonly used in cases where classes are unbalanced - to avoid, for example, folds where there are no (or very few, or just inefficiently few) positive examples in a sparse binary classification problem. This applies both to StratifiedShuffleSplit and StratifiedKFold methods.

In order to determine which to use between K-Fold and ShuffleSplit, though, we'll have to understand some key differences between the methods.

  1. In K-fold, the model is trained on each iteration at a proportion of the data set equal to $\frac{k-1}{k}$, i.e. for $k=5$ the model is trained on 80% of the data at each iteration and for $k=10$ the model is trained on 90% of the data. There are $k$ training iterations in the algorithm.

    In ShuffleSplit, the model is trained at each iteration on a defined train_size. The default size of the training set for the scikit-learn implementation is 90%. The number of iterations is parameterized (n_splits).

    You can therefore configure two validation strategies that have the same number of training runs and are trained on the same proportion of the data, I.e. K-fold validation with K=10 and ShuffleSplit with train_size = .9 and n_splits = 10.

No difference here.

  1. K-Fold is guaranteed to both train and test on every observation of the dataset an equal number of times ($k-1$ and 1 times, respectively).

    ShuffleSplit does not guarantee this proportion- by re-sampling at each iteration, duplicate members of the test set can be selected twice, or several times. Some observations may not be represented in the test set.

This is a point for using K-Fold.

Some other discussion has brought up training time as a point in favor of ShuffleSplit- that is, you can configure ShuffleSplit to run with less demanding coverage (i.e. train_size = .8, n_splits = 5). This is only a strong argument for complexities that can't be approximated by adjusting the parameter $k$- as $k=5$ for the parameters mentioned above for ShuffleSplit.

For example, K-fold can't approximate train_size = .8, n_splits = 3. The tradeoff is that each configuration of ShuffleSplit that would be more computationally efficient than K-Fold also provides less complete validation coverage- in this example (train_size = .8, n_splits = 3), the model could never be tested on more than 60% of the data, and would almost always be tested on less than that.

K-Fold is generally better except in edge cases where the computational demand of K-Fold isn't supported by a need for comprehensive cross-validation coverage.

  • $\begingroup$ Just for clarification then, going by what you said above. Is K-Fold equivalent to ShuffleSplit validation if for the latter it is shown that we are are generating new folds without replacement after each fold has been tested? (Is Shuffle-Split implemented that way?) $\endgroup$ – Thornhale Mar 29 '17 at 23:45
  • 1
    $\begingroup$ No, these two are not equivalent- ShuffleSplit is basically just iterations of train/test splitting. I.e. in iteration 1, you train/test split on the whole set, in iteration 2 you re-split the set and test again etc. What I was trying to communicate was unrelated to ShuffleSplit directly- basically, if you form the k folds sequentially rather than all at once, it's equivalent to doing it all at once. That statement just covers one possible interpretation of the question. $\endgroup$ – Thomas Cleberg Mar 30 '17 at 13:20
  • $\begingroup$ Ah, ok. That part is clear. The only part that is not clear to me then is: When should I prefer ShuffleSplit over Kfold? And when should I prefer Kfold over SHuffleSplit? $\endgroup$ – Thornhale Mar 30 '17 at 13:27
  • 1
    $\begingroup$ Added more to clarify the tradeoffs of this decision. $\endgroup$ – Thomas Cleberg Mar 30 '17 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.