4
$\begingroup$

Suppose we have a dataset $D\in\mathbb{R}^{n\times p}$, where $n = 60000$ samples/observations and $p = 3072$ variables. To my understanding one would perform cross validation if found in one of the following scenarios:

1.) model evaluation (my use case scenario)

2.) model selection

3.) model comparison

My approach involves splitting $D$ into $90\%/10\%$ as $D_{train}$ and $D_{test}$ equivalently. Then performing 10-fold stratified CV on $D_{train}$ which is further subdivided into $D_{learn}$ and $D_{valid}$ after every fold in order to get an estimate of model's generalisation performance and finally test on the untouched $D_{test}$ to get the final performance estimate.

After every fold $k$ the model is trained on $D_{learn}$ for 100 epochs and evaluated on $D_{valid}$. The average performance on $D_{valid}$ over the $k$ folds is 62% and 90% on $D_{learn}$. Finally, I evaluate the model's performance on the untouched test set $D_{test}$ to see if it agrees with the average performance on the $D_{valid}$ after the $k$ folds, and the results are pretty bad leading to 22% accuracy on $D_{test}$ and an average 62% accuracy on $D_{valid}$ which brings me to the following questions.

Q1) Is my approach valid?

Q2) I've read that sometimes is appropriate to just split the dataset into train/validation/test if the dataset is sufficiently large. What does sufficiently large mean and how do we define it or understand that our dataset falls under this scenario?

Q3) Is it normal to account such divergence between $D_{learn}$ and $D_{valid}$ during the folds?

Q4) Could the divergence in performance between average accuracy on the 10 folds and final test set ($D_{valid}$ vs $D_{test}$) be because I don't perform stratified cross validation on $D_{test}$?

Q5) When and under which condition it is necessary and sufficient to perform cross validation on the test set as well as on the train set?

Thanks!

$\endgroup$
  • $\begingroup$ "Finally, I evaluate the model's performance on the untouched test set" - which model? You're not accidentally continuing training a single model over each fold, are you? Also, I don't understand Q5. $\endgroup$ – Ben Reiniger Aug 22 at 21:06
  • $\begingroup$ Hi @BenReiniger, thanks for the comment. I'm not quite sure whether it would make any difference disclosing which model, I suppose I wrote the questions with no specific model in mind. But if it helps you contextualize the problem we can think of the dataset being composed of images and the model being just a classifier on a multi-class problem. Q5) Is asking in a way if you perform stratified CV on the 90% of data, should you also do it on the remaining 10% which is used to be the untouched test set? $\endgroup$ – kirk Aug 22 at 23:07
  • $\begingroup$ Sorry, I didn't mean what type of model. I meant, in your k-fold CV, you train k different models. Do you then train another model on the entire train set? (And hence the following question: make sure you're training on each subsequent fold from scratch, not a continuation of training.) I'm not sure how you intend to do CV on the test set? $\endgroup$ – Ben Reiniger Aug 23 at 2:34
  • 1
    $\begingroup$ Hi @BenReiniger, yup on each fold I start with a new model where all parameters are randomly initialized. Out of k different models I only keep the best performing one. I don't train the best performing one again on the whole dataset I just evaluate it on the untouched test set. Regarding the test set I've read that on some occasions when you perform stratified cv on the train set you can do the same on the test set as well but the details are fuzzy to me as well regarding the appropriate conditions. $\endgroup$ – kirk Aug 23 at 2:44
  • 1
    $\begingroup$ Hi @BenReiniger, you are right the 62% is the average score on the validation split across k folds. Initially I would have thought of the data distribution being imbalanced but this particular dataset is fairly known to the ML community and seems balanced overall. The dataset comes split as 90%/10% with equivalent samples of 50k for train 10k for test. The total number of classes is 10. The test set contains 1000 randomly-selected images from each class, in total 1kx10=10k. The training set contain exactly 5000 images from each class, 5kx10=50k. $\endgroup$ – kirk Aug 23 at 10:10
3
$\begingroup$

Q1) Is my approach valid?

Yes, it is a pretty standard strategy.

Q2) I've read that sometimes is appropriate to just split the dataset into train/validation/test if the dataset is sufficiently large. What does sufficiently large mean and how do we define it or understand that our dataset falls under this scenario?

There aren't specific numbers to go by. The problem with CV is that it requires the model to be trained $k$ times. If your dataset is large and training takes a lot of time, you might not have the luxury of performing CV. So you turn to a hold out strategy.

An example of this is image classification. Large CNNs can take weeks to be trained. In this domain. If you were to perform lets say a 5-fold CV here, something that would take 3 weeks to complete requires nearly 4 months!

Q3) Is it normal to account such divergence between $D_{learn}$ and $D_{valid}$ during the folds?

No, its a sign of overfitting. You can check this post on ways of preventing this.

Q4) Could the divergence in performance between average accuracy on the 10 folds and final test set ($D_{valid}$ vs $D_{test}$) be because I don't perform stratified cross validation on $D_{test}$ ?

What do you mean cross-validation on the test set? Isn't it a standard hold-out test set?

The performance difference between $D_{valid}$ vs $D_{test}$, is because you have overfit on the validation set

Q5) When and under which condition it is necessary and sufficient to perform cross validation on the test set as well as on the train set?

I don't get what you mean "cross-validation on the test set". Do you mean to split the test into k-folds and train the model k times each time evaluating on a different fold etc.? I don't get the meaning of this because you already did this on the train/validation set.

In your case the test set is fulfilling its purpose: it showed you that you (significantly) overfit! You shouldn't change your experimental procedure, rather your model and choice hyperparameters so that you don't overfit.

$\endgroup$
  • $\begingroup$ thanks for the answers, what I mean by CV on the test is the following, the initial split of train/test might have created unequal distribution of samples for each class on the test set and performing stratified CV on train and evaluating on test could yield this divergence of score, hence the question of performing stratified CV on test and averaging the score and using that to get the difference between the average score of stratified CV on train vs the same on test. $\endgroup$ – kirk Aug 24 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.