39
$\begingroup$

In some lectures and tutorials I've seen, they suggest to split your data into three parts: training, validation and test. But it is not clear how the test dataset should be used, nor how this approach is better than cross-validation over the whole data set.

Let's say we have saved 20% of our data as a test set. Then we take the rest, split it into k folds and, using cross-validation, we find the model that makes the best prediction on unknown data from this dataset. Let's say the best model we have found gives us 75% accuracy.

Various tutorials and lots of questions on various Q&A websites say that now we can verify our model on a saved (test) dataset. But I still can't get how exactly is it done, nor what is the point of it.

Let's say we've got an accuracy of 70% on the test dataset. So what do we do next? Do we try another model, and then another, until we will get a high score on our test dataset? But in this case it really looks like we will just find the model that fits our limited (only 20%) test set. It doesn't mean that we will find the model that is best in general.

Moreover, how can we consider this score as a general evaluation of the model, if it is only calculated on a limited data set? If this score is low, maybe we were unlucky and selected "bad" test data.

On the other hand, if we use all the data we have and then choose the model using k-fold cross-validation, we will find the model that makes the best prediction on unknown data from the entire data set we have.

$\endgroup$
5
  • 1
    $\begingroup$ You evaluate your best model on that test set and report the performance on it. It is your best estimate of the performance of your model. I suggest you to read the first two chapters of excellent book "Learning from Data" by Yaser Abu-Mostafa. Very succinct and very accessible. work.caltech.edu/telecourse.html $\endgroup$ – Vladislavs Dovgalecs May 18 '15 at 21:08
  • 1
    $\begingroup$ Thanks for the book you've suggested! But regarding your answer to the question - you say it is the "best estimate of the performance of your model", but it is actually the estimate of the performance of a model on a small(20%) test set, it doesn't mean the model the performance in general. $\endgroup$ – Serhiy May 18 '15 at 21:18
  • 2
    $\begingroup$ Actually it is your best estimate of the model's performance in general. Your model might be biased and/or suffer from high variance, but the model's performance on your test set is the best indicator how it would perform on any unseen data. $\endgroup$ – Vladislavs Dovgalecs May 19 '15 at 17:28
  • 1
    $\begingroup$ Thanks xeon! The only thing that is still unclear to me, is what do we do after assessing the model using test data set? $\endgroup$ – Serhiy May 19 '15 at 18:07
  • $\begingroup$ You either ship the model if you are happy with the results or find better features/algorithm/more data to improve the model. $\endgroup$ – Vladislavs Dovgalecs May 19 '15 at 18:09
27
$\begingroup$

This is similar to another question I answered regarding cross-validation and test sets. The key concept to understand here is independent datasets. Consider just two scenarios:

  1. If you have lot's of resources you would ideally collect one dataset and train your model via cross-validation. Then you would collect another completely independent dataset and test your model. However, as I said previously, this is usually not possible for many researchers.

Now, if I am a researcher who isn't so fortunate what do I do? Well, you can try to mimic that exact scenario:

  1. Before you do any model training you would take a split of your data and leave it to the side (never to be touched during cross-validation). This is to simulate that very same independent dataset mentioned in the ideal scenario above. Even though it comes from the same dataset the model training won't take any information from those samples (where with cross-validation all the data is used). Once you have trained your model you would then apply it to your test set, again that was never seen during training, and get your results. This is done to make sure your model is more generalizable and hasn't just learned your data.

To address your other concerns:

Let's say we've got an accuracy of 70% on test data set, so what do we do next? Do we try an other model, and then an other, untill we will get hight score on our test data set?

Sort of, the idea is that you are creating the best model you can from your data and then evaluating it on some more data it has never seen before. You can re-evaluate your cross-validation scheme but once you have a tuned model (i.e. hyper parameters) you are moving forward with that model because it was the best you could make. The key is to NEVER USE YOUR TEST DATA FOR TUNING. Your result from the test data is your model's performance on 'general' data. Replicating this process would remove the independence of the datasets (which was the entire point). This is also address in another question on test/validation data.

And also, how can we consider this score as general evaluation of the model, if it is calculated on a limited data set? If this score is low, maybe we were unlucky to select "bad" test data.

This is unlikely if you have split your data correctly. You should be splitting your data randomly (although potentially stratified for class balancing). If you dataset is large enough that you are splitting your data in to three parts, your test subset should be large enough that the chance is very low that you just chose bad data. It is more likely that your model has been overfit.

$\endgroup$
17
  • 7
    $\begingroup$ Thank you for such detailed explanation! The only thing that is still unclear to me, is what do we do after assessing the model using test data set? What if the result is really low? Do we try an other model? $\endgroup$ – Serhiy May 19 '15 at 18:06
  • 1
    $\begingroup$ As I say above, you can re-evaluate your cross-validation and see if your method can be improved so long as you don't use your 'test' data for model training. If your result is low you likely have overfit your model. Your dataset may only have so much predictive power. $\endgroup$ – cdeterman May 19 '15 at 18:39
  • 2
    $\begingroup$ @Zelphir no, each fold is evaluated multiple times with each hyperparameter combination. Simply choosing the best result is likely to report higher results than on an 'independent' test set. The idea is to see which parameters generalize across these folds and use that architecture on a test set it has never seen to determine how 'generalizable' the model is. $\endgroup$ – cdeterman Nov 21 '16 at 16:48
  • 1
    $\begingroup$ @KevinKim, good question. There is a chance that the model with the best CV score may not have the best score on the test set. However, the best CV model, if built correctly, should have the best generalization (i.e. best over many other test sets). This is because your CV has evaluated a model against different subsets. Now, there is no perfect method, no way to guarantee your model configuration is the best. CV is just a means of controlling for overfitting. CV is a trustworthy criteria but just an estimate, not an absolute. $\endgroup$ – cdeterman Mar 2 '18 at 15:44
  • 3
    $\begingroup$ I want to add that actually the act of looking at performance on your independent test set and deciding to go back and retrain your model makes your test set no longer completely independent. In fact you could do this infinitely many times until you get perfect accuracy. In regards to academic reference, I recommend the book "Learning from data", there is also free online course by the author (work.caltech.edu/telecourse.html). $\endgroup$ – Michael Aug 28 '19 at 9:14
5
$\begingroup$

If all you are going to do is train a model with default settings on the raw or minimally preprocessed dataset (e.g. one-hot encoding and/or removing NAs), you don't need a separate test set, you can simply train on your train set and test on your validation set, or even better, train on the entire set using cross-validation to estimate your performance.

However, as soon as your knowledge about the data causes you to make any changes from your original strategy, you have now "tainted" your result. Some examples include:

  • Model choice: You tested logistic, lasso, random forest, XGBoost, and support vector machines and choose the best model

  • Parameter tuning: You tuned an XGBoost to find the optimal hyperparameters

  • Feature selection: You used backward selection, genetic algorithm, boruta, etc. to choose an optimal subset of features to include in your model

  • Missing imputation: You imputed missing variables with the mean, or with a simple model based on the other variables

  • Feature transformation: You centered and scaled your numeric variables to replace them with a z-score (number of standard deviations from the mean)

In all of the above cases, using a single holdout set, or even cross-validation, is not going to give you a realistic estimate of real-world performance because you are using information you won't have on future data in your decision. Instead, you are cherry-picking the best model, the best hyperparameters, the best feature set, etc. for your data, and you are likely to be slightly "overfitting" you strategy to your data. To get an honest estimate of real-world performance, you need to score it on data that didn't enter into the decision process at all, hence the common practice of using an independent test set separate from your training (modeling) and validation (picking a model, features, hyperparameters, etc.) set.

As an alternative to holding out a test set, you can instead use a technique called nested cross-validation. This requires you to code up your entire modeling strategy (transformation, imputation, feature selection, model selection, hyperparameter tuning) as a non-parametric function and then perform cross-validation on that entire function as if it were simply a model fit function. This is difficult to do in most ML packages, but can be implemented quite easily in R with the mlr package by using wrappers to define your training strategy and then resampling your wrapped learner:

https://mlr.mlr-org.com/articles/tutorial/nested_resampling.html

$\endgroup$
1
$\begingroup$

I'm assuming that you're doing classification.

Take your data and split it 70/30 into trainingData/ testData subsets. Take the trainingData subset and split it 70/30 again into trainingData/ validateData subsets. Now you have 3 subsets of your original data - trainingData(.7*.7), validateData(.7*.3), and testData(.3).

You train your model with trainingData. Then, you check that model's performance using validateData, which we can think of as independent of trainingData and therefore a good evaluation of how well the model is generalizing. Let's pretend that you achieve 75% accuracy.

Now you retrain your model an arbitrary number of times. Each retraining, you're evaluating a different set of hyperparameters (the parameters being fed to your model in the first place vs those your model is optimizing for) but still using the trainingData subset. Each retraining, you're also again checking how well the new model generalizes by checking performance on validateData.

Once you've checked every combination of hyperparameters you mean to assess, you choose the set of hyperparameters that gave you your best performance on validateData - let's pretend your best performance on validateData was 80% accuracy. These are your final hyperparameters and the model defined by those hyperparameters is the one you'll use for this next step.

Now you take the model that uses your final hyperparameters and evaluate testData. This is the first time testData has been touched since this whole process started! If you get testData performance that's comparable to your performance on validateData (although usually it will be slightly lower), then you can feel confident that your model works as expected and generalizes well! If that happens, this is your final model!

Why do all of this? You're trying to avoid overfitting. There's always a risk that you are overfitting to the data you use when you're training and tuning (aka validating) your model. If you train, tune (validate), and test using just one data set, there's a good chance you'll overfit that data and it won't generalize well. By breaking training and test data sets apart (and assuming you tune using the test data), you have the chance to check yourself internally, but there's still the chance that you're just overfitting the test data now. That's why we break out a third data set, validate, so we have an additional layer of keeping ourselves internally honest. Tuning with validateData keeps us from overfitting to trainingData. Final testing with testData keeps us from overfitting to validateData.

$\endgroup$
3
  • $\begingroup$ How is this answer different from the accepted answer? $\endgroup$ – Jan Kukacka Mar 5 '18 at 10:42
  • $\begingroup$ The original question is explicitly asking about a workflow that doesn't include cross validation. I read his question to be asking how it works and why it's reliable. The accepted answer discusses a workflow that still includes cross validation. I wanted to explain how people do it without necessarily using cross validation in case somebody is a beginner and maybe isn't at that point, yet. Sorry if posting this is wrong - I am a new user and don't mean to have broken a rule. $\endgroup$ – John D Mar 5 '18 at 21:27
  • $\begingroup$ But how can we do hyperparameter optimization this way? We train on training data, validate on validation data, build a model with training + validation data and test it on test data - but what do we do if the accuracy / RMSE is low? If we optimize with training+validation data to improve accuracy / RMSE on the test data we optimize on the test data in the end? Which makes it basically the same as validation data? $\endgroup$ – Vega Jul 16 '20 at 8:53
1
$\begingroup$

Let us look at it the following way

  1. Common practice

    a) Training data - used for choosing model parameters.

     i) E.g., finding intercept and slope parameters for an ordinary linear 
        regression model. 
    
     ii) The noise in the training data-set is used in some extent 
         in over-fitting model parameters. 
    

    b) Validation data - used for choosing hyper-parameters.

     i)  E.g., we may want to test three different models at step 1.a, say 
         linear model with one, two or three variables.   
    
     ii) The validation data-set is independent from training data, and thus, they provide 
         'unbiased' evaluation to the models, which help to decide which 
         hyper-parameter to use. 
    
     iii) We note that, a model trained in 1.a, say y = b_0+b_1*x_1, does 
         not learn anything from this data-set. So, the noise in this data- 
         set is not used to over-fit the parameters (b_0, b_1), but, over- 
         fit exists in choosing which linear model to use (in terms of 
         number of variables). 
    

    c) Test data - used to get confidence of the output from the above two steps

    i) Used once a model is completely trained
    
  2. Another way to look at part 1

    a) Our model candidate pool is a 5-dimenson set, i.e.,

    i) Dimension 1: number of variables to keep in the regression model, 
       e.g., [1, 2, 3].
    
    ii) Dimension 2-5: (b_0, b_1, b_2, b_3). 
    

    b) Step 1a reduce model candidates from 5-dimension to 1-dimension.

    c) Step 1b reduce model candidates from 1-dimension to 0-dimension, which a single model.

    d) However, the OP may think the ‘final’ output above is not performing well enough on the test data set, and thus redo the whole process again, say using ridge regression instead of ordinary linear regression. Then the test data set is used multiple times and thus the noise in this data might produce some overfitting in deciding whether to use linear regression or ridge regression.

    e) To deal with a high dimensional model pool with parameters, hyperparameters, model types and pre-processing methods, any split to the data available to us is essentially defining a decision-making process which

    i)  Sequentially reducing the model pool to zero-dimension.
    
    ii) Allocating data noise overfitting to different steps of dimension 
        reductions (overfitting the noise in the data is not avoidable but 
        could be allocated smartly). 
    
  3. Conclusion and answers to OP’s question

    a) Two-split (training and test), three-split (training, validating and testing) or higher number of split are essentially about reducing dimensionality and allocating the data (especially noise and risk of over- fitting).

    b) At some stage, you may come up a ‘final’ model candidate pool, and then, you can think of how to design the process of reducing the dimension sequentially such that

    i) At each step of reducing the dimensions, the output is satisfactory, 
      e.g., not using just 10 data points with large noise to estimate a 
      six-parameter liner model. 
    
    ii) There are enough data for you to reduce the dimension to zero 
       finally. 
    

    c) What if you cannot achieve b

    i) Use model and data insight to reduce the overall dimensionality of 
      your model pool. E.g., liner regression is sensitive to outliers thus 
      not good for data with many large outliers. 
    
    ii) Choose robust non-parametric models or models with less number of 
       parameter if possible. 
    
    iii) Smartly allocating the data available at each step of reducing the 
        dimensionality. There is some goodness of fit tests to help us decide 
        whether the data we use to train the model is enough or not. 
    
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.