How do you use the 'test' dataset after cross-validation?

Question

In some lectures and tutorials I've seen, they suggest to split your data into three parts: training, validation and test. But it is not clear how the test dataset should be used, nor how this approach is better than cross-validation over the whole data set.

Let's say we have saved 20% of our data as a test set. Then we take the rest, split it into k folds and, using cross-validation, we find the model that makes the best prediction on unknown data from this dataset. Let's say the best model we have found gives us 75% accuracy.

Various tutorials and lots of questions on various Q&A websites say that now we can verify our model on a saved (test) dataset. But I still can't get how exactly is it done, nor what is the point of it.

Let's say we've got an accuracy of 70% on the test dataset. So what do we do next? Do we try another model, and then another, until we will get a high score on our test dataset? But in this case it really looks like we will just find the model that fits our limited (only 20%) test set. It doesn't mean that we will find the model that is best in general.

Moreover, how can we consider this score as a general evaluation of the model, if it is only calculated on a limited data set? If this score is low, maybe we were unlucky and selected "bad" test data.

On the other hand, if we use all the data we have and then choose the model using k-fold cross-validation, we will find the model that makes the best prediction on unknown data from the entire data set we have.

Answer 1

This is similar to another question I answered regarding cross-validation and test sets. The key concept to understand here is independent datasets. Consider just two scenarios:

1. If you have lot's of resources you would ideally collect one dataset and train your model via cross-validation. Then you would collect another completely independent dataset and test your model. However, as I said previously, this is usually not possible for many researchers.

Now, if I am a researcher who isn't so fortunate what do I do? Well, you can try to mimic that exact scenario:

2. Before you do any model training you would take a split of your data and leave it to the side (never to be touched during cross-validation). This is to simulate that very same independent dataset mentioned in the ideal scenario above. Even though it comes from the same dataset the model training won't take any information from those samples (where with cross-validation all the data is used). Once you have trained your model you would then apply it to your test set, again that was never seen during training, and get your results. This is done to make sure your model is more generalizable and hasn't just learned your data.

To address your other concerns:

Let's say we've got an accuracy of 70% on test data set, so what do we do next? Do we try an other model, and then an other, untill we will get hight score on our test data set?

Sort of, the idea is that you are creating the best model you can from your data and then evaluating it on some more data it has never seen before. You can re-evaluate your cross-validation scheme but once you have a tuned model (i.e. hyper parameters) you are moving forward with that model because it was the best you could make. The key is to NEVER USE YOUR TEST DATA FOR TUNING. Your result from the test data is your model's performance on 'general' data. Replicating this process would remove the independence of the datasets (which was the entire point). This is also address in another question on test/validation data.

And also, how can we consider this score as general evaluation of the model, if it is calculated on a limited data set? If this score is low, maybe we were unlucky to select "bad" test data.

This is unlikely if you have split your data correctly. You should be splitting your data randomly (although potentially stratified for class balancing). If you dataset is large enough that you are splitting your data in to three parts, your test subset should be large enough that the chance is very low that you just chose bad data. It is more likely that your model has been overfit.

Answer 2

If all you are going to do is train a model with default settings on the raw or minimally preprocessed dataset (e.g. one-hot encoding and/or removing NAs), you don't need a separate test set, you can simply train on your train set and test on your validation set, or even better, train on the entire set using cross-validation to estimate your performance.

However, as soon as your knowledge about the data causes you to make any changes from your original strategy, you have now "tainted" your result. Some examples include:

• Model choice: You tested logistic, lasso, random forest, XGBoost, and support vector machines and choose the best model

• Parameter tuning: You tuned an XGBoost to find the optimal hyperparameters

• Feature selection: You used backward selection, genetic algorithm, boruta, etc. to choose an optimal subset of features to include in your model

• Missing imputation: You imputed missing variables with the mean, or with a simple model based on the other variables

• Feature transformation: You centered and scaled your numeric variables to replace them with a z-score (number of standard deviations from the mean)

In all of the above cases, using a single holdout set, or even cross-validation, is not going to give you a realistic estimate of real-world performance because you are using information you won't have on future data in your decision. Instead, you are cherry-picking the best model, the best hyperparameters, the best feature set, etc. for your data, and you are likely to be slightly "overfitting" you strategy to your data. To get an honest estimate of real-world performance, you need to score it on data that didn't enter into the decision process at all, hence the common practice of using an independent test set separate from your training (modeling) and validation (picking a model, features, hyperparameters, etc.) set.

As an alternative to holding out a test set, you can instead use a technique called nested cross-validation. This requires you to code up your entire modeling strategy (transformation, imputation, feature selection, model selection, hyperparameter tuning) as a non-parametric function and then perform cross-validation on that entire function as if it were simply a model fit function. This is difficult to do in most ML packages, but can be implemented quite easily in R with the mlr package by using wrappers to define your training strategy and then resampling your wrapped learner:

https://mlr.mlr-org.com/articles/tutorial/nested_resampling.html

Answer 3

I'm assuming that you're doing classification.

Take your data and split it 70/30 into trainingData/ testData subsets. Take the trainingData subset and split it 70/30 again into trainingData/ validateData subsets. Now you have 3 subsets of your original data - trainingData(.7*.7), validateData(.7*.3), and testData(.3).

You train your model with trainingData. Then, you check that model's performance using validateData, which we can think of as independent of trainingData and therefore a good evaluation of how well the model is generalizing. Let's pretend that you achieve 75% accuracy.

Now you retrain your model an arbitrary number of times. Each retraining, you're evaluating a different set of hyperparameters (the parameters being fed to your model in the first place vs those your model is optimizing for) but still using the trainingData subset. Each retraining, you're also again checking how well the new model generalizes by checking performance on validateData.

Once you've checked every combination of hyperparameters you mean to assess, you choose the set of hyperparameters that gave you your best performance on validateData - let's pretend your best performance on validateData was 80% accuracy. These are your final hyperparameters and the model defined by those hyperparameters is the one you'll use for this next step.

Now you take the model that uses your final hyperparameters and evaluate testData. This is the first time testData has been touched since this whole process started! If you get testData performance that's comparable to your performance on validateData (although usually it will be slightly lower), then you can feel confident that your model works as expected and generalizes well! If that happens, this is your final model!

Why do all of this? You're trying to avoid overfitting. There's always a risk that you are overfitting to the data you use when you're training and tuning (aka validating) your model. If you train, tune (validate), and test using just one data set, there's a good chance you'll overfit that data and it won't generalize well. By breaking training and test data sets apart (and assuming you tune using the test data), you have the chance to check yourself internally, but there's still the chance that you're just overfitting the test data now. That's why we break out a third data set, validate, so we have an additional layer of keeping ourselves internally honest. Tuning with validateData keeps us from overfitting to trainingData. Final testing with testData keeps us from overfitting to validateData.

Answer 4

Let us look at it the following way

1. Common practice

   a) Training data - used for choosing model parameters.

    i) E.g., finding intercept and slope parameters for an ordinary linear 
       regression model. 
   
    ii) The noise in the training data-set is used in some extent 
        in over-fitting model parameters. 
   

   b) Validation data - used for choosing hyper-parameters.

    i)  E.g., we may want to test three different models at step 1.a, say 
        linear model with one, two or three variables.   
   
    ii) The validation data-set is independent from training data, and thus, they provide 
        'unbiased' evaluation to the models, which help to decide which 
        hyper-parameter to use. 
   
    iii) We note that, a model trained in 1.a, say y = b_0+b_1*x_1, does 
        not learn anything from this data-set. So, the noise in this data- 
        set is not used to over-fit the parameters (b_0, b_1), but, over- 
        fit exists in choosing which linear model to use (in terms of 
        number of variables). 
   

   c) Test data - used to get confidence of the output from the above two steps

   i) Used once a model is completely trained
   

2. Another way to look at part 1

   a) Our model candidate pool is a 5-dimenson set, i.e.,

   i) Dimension 1: number of variables to keep in the regression model, 
      e.g., [1, 2, 3].
   
   ii) Dimension 2-5: (b_0, b_1, b_2, b_3). 
   

   b) Step 1a reduce model candidates from 5-dimension to 1-dimension.

   c) Step 1b reduce model candidates from 1-dimension to 0-dimension, which a single model.

   d) However, the OP may think the ‘final’ output above is not performing well enough on the test data set, and thus redo the whole process again, say using ridge regression instead of ordinary linear regression. Then the test data set is used multiple times and thus the noise in this data might produce some overfitting in deciding whether to use linear regression or ridge regression.

   e) To deal with a high dimensional model pool with parameters, hyperparameters, model types and pre-processing methods, any split to the data available to us is essentially defining a decision-making process which

   i)  Sequentially reducing the model pool to zero-dimension.
   
   ii) Allocating data noise overfitting to different steps of dimension 
       reductions (overfitting the noise in the data is not avoidable but 
       could be allocated smartly). 
   

3. Conclusion and answers to OP’s question

   a) Two-split (training and test), three-split (training, validating and testing) or higher number of split are essentially about reducing dimensionality and allocating the data (especially noise and risk of over- fitting).

   b) At some stage, you may come up a ‘final’ model candidate pool, and then, you can think of how to design the process of reducing the dimension sequentially such that

   i) At each step of reducing the dimensions, the output is satisfactory, 
     e.g., not using just 10 data points with large noise to estimate a 
     six-parameter liner model. 
   
   ii) There are enough data for you to reduce the dimension to zero 
      finally. 
   

   c) What if you cannot achieve b

   i) Use model and data insight to reduce the overall dimensionality of 
     your model pool. E.g., liner regression is sensitive to outliers thus 
     not good for data with many large outliers. 
   
   ii) Choose robust non-parametric models or models with less number of 
      parameter if possible. 
   
   iii) Smartly allocating the data available at each step of reducing the 
       dimensionality. There is some goodness of fit tests to help us decide 
       whether the data we use to train the model is enough or not.