When and under what conditions would anyone perform cross validation on the test set?

Question

Suppose we have a dataset $D\in\mathbb{R}^{n\times p}$, where $n = 60000$ samples/observations and $p = 3072$ variables. To my understanding one would perform cross validation if found in one of the following scenarios:

1.) model evaluation (my use case scenario)

2.) model selection

3.) model comparison

My approach involves splitting $D$ into $90\%/10\%$ as $D_{train}$ and $D_{test}$ equivalently. Then performing 10-fold stratified CV on $D_{train}$ which is further subdivided into $D_{learn}$ and $D_{valid}$ after every fold in order to get an estimate of model's generalisation performance and finally test on the untouched $D_{test}$ to get the final performance estimate.

After every fold $k$ the model is trained on $D_{learn}$ for 100 epochs and evaluated on $D_{valid}$. The average performance on $D_{valid}$ over the $k$ folds is 62% and 90% on $D_{learn}$. Finally, I evaluate the model's performance on the untouched test set $D_{test}$ to see if it agrees with the average performance on the $D_{valid}$ after the $k$ folds, and the results are pretty bad leading to 22% accuracy on $D_{test}$ and an average 62% accuracy on $D_{valid}$ which brings me to the following questions.

Q1) Is my approach valid?

Q2) I've read that sometimes is appropriate to just split the dataset into train/validation/test if the dataset is sufficiently large. What does sufficiently large mean and how do we define it or understand that our dataset falls under this scenario?

Q3) Is it normal to account such divergence between $D_{learn}$ and $D_{valid}$ during the folds?

Q4) Could the divergence in performance between average accuracy on the 10 folds and final test set ($D_{valid}$ vs $D_{test}$) be because I don't perform stratified cross validation on $D_{test}$?

Q5) When and under which condition it is necessary and sufficient to perform cross validation on the test set as well as on the train set?

Thanks!

Answer 1

Q1) Is my approach valid?

Yes, it is a pretty standard strategy.

Q2) I've read that sometimes is appropriate to just split the dataset into train/validation/test if the dataset is sufficiently large. What does sufficiently large mean and how do we define it or understand that our dataset falls under this scenario?

There aren't specific numbers to go by. The problem with CV is that it requires the model to be trained $k$ times. If your dataset is large and training takes a lot of time, you might not have the luxury of performing CV. So you turn to a hold out strategy.

An example of this is image classification. Large CNNs can take weeks to be trained. In this domain. If you were to perform lets say a 5-fold CV here, something that would take 3 weeks to complete requires nearly 4 months!

Q3) Is it normal to account such divergence between $D_{learn}$ and $D_{valid}$ during the folds?

No, its a sign of overfitting. You can check this post on ways of preventing this.

Q4) Could the divergence in performance between average accuracy on the 10 folds and final test set ($D_{valid}$ vs $D_{test}$) be because I don't perform stratified cross validation on $D_{test}$ ?

What do you mean cross-validation on the test set? Isn't it a standard hold-out test set?

The performance difference between $D_{valid}$ vs $D_{test}$, is because you have overfit on the validation set

Q5) When and under which condition it is necessary and sufficient to perform cross validation on the test set as well as on the train set?

I don't get what you mean "cross-validation on the test set". Do you mean to split the test into k-folds and train the model k times each time evaluating on a different fold etc.? I don't get the meaning of this because you already did this on the train/validation set.

In your case the test set is fulfilling its purpose: it showed you that you (significantly) overfit! You shouldn't change your experimental procedure, rather your model and choice hyperparameters so that you don't overfit.