I found this confusing when I use the neural network toolbox in Matlab.
It divided the raw data set into three parts:

  1. training set
  2. validation set
  3. test set

I notice in many training or learning algorithm, the data is often divided into 2 parts, the training set and the test set.

My questions are:

  1. what is the difference between validation set and test set?
  2. Is the validation set really specific to neural network? Or it is optional.
  3. To go further, is there a difference between validation and testing in context of machine learning?
  • 50
    The question is answered in the book Elements of statistical learning page 222. The validation set is used for model selection, the test set for final model (the model which was selected by selection process) prediction error. – mpiktas Nov 28 '11 at 11:47
  • @mpiktas Are you referring to the chapter "Model Assessment and Selection"? – Celdor Jun 1 '15 at 7:16
  • 2
    Yes. The page number was from 5th print edition. – mpiktas Jun 1 '15 at 7:20
  • You might want to also see: stats.stackexchange.com/questions/9357/…, where the question was "Why not more than three?" – Wayne Sep 29 '15 at 14:28
  • 11
    @mpiktas is spot on. Here is the actual text: The training set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model. Ideally, the test set should be kept in a “vault,” and be brought out only at the end of the data analysis. – arun Jul 15 '16 at 18:01

11 Answers 11

up vote 231 down vote accepted

Normally to perform supervised learning you need two types of data sets:

  1. In one dataset (your "gold standard") you have the input data together with correct/expected output, This dataset is usually duly prepared either by humans or by collecting some data in semi-automated way. But it is important that you have the expected output for every data row here, because you need this for supervised learning.

  2. The data you are going to apply your model to. In many cases this is the data in which you are interested for the output of your model and thus you don't have any "expected" output here yet.

While performing machine learning you do the following:

  1. Training phase: you present your data from your "gold standard" and train your model, by pairing the input with expected output.
  2. Validation/Test phase: in order to estimate how well your model has been trained (that is dependent upon the size of your data, the value you would like to predict, input etc) and to estimate model properties (mean error for numeric predictors, classification errors for classifiers, recall and precision for IR-models etc.)
  3. Application phase: now you apply your freshly-developed model to the real-world data and get the results. Since you normally don't have any reference value in this type of data (otherwise, why would you need your model?), you can only speculate about the quality of your model output using the results of your validation phase.

The validation phase is often split into two parts:

  1. In the first part you just look at your models and select the best performing approach using the validation data (=validation)
  2. Then you estimate the accuracy of the selected approach (=test).

Hence the separation to 50/25/25.

In case if you don't need to choose an appropriate model from several rivaling approaches, you can just re-partition your set that you basically have only training set and test set, without performing the validation of your trained model. I personally partition them 70/30 then.

See also this question.

  • 16
    Why wouldn't I choose the best performing model based on the test set, getting rid of the validation set altogether? – Sebastian Graf Nov 9 '14 at 14:31
  • 2
    Is it because of overfitting? Or because we want some independent statistics based on the test result, just for error estimation? – Sebastian Graf Nov 9 '14 at 14:42
  • 12
    @Sebastian [If you only use the test set: ]"The test set error of the final chose model will underestimate the true test error, sometimes significantly" [Hastie et al] – user695652 Jun 2 '15 at 20:09
  • 20
    The validation set is often used to tune hyper-parameters. For example, in the deep learning community, tuning the network layer size, hidden unit number, regularization term(wether L1 or L2) depends on the validation set – xiaohan2012 Oct 13 '15 at 10:52
  • 2
    What is the correct way to split the sets? Should the selection be random? What if you have pictures that are similar? Won't this damage your ability to generalize? If you have two sets taken in separate locations wouldn't it be better to take one as training set and the other as the test set? – Yonatan Simson Feb 3 '16 at 10:36

Training set: a set of examples used for learning: to fit the parameters of the classifier In the MLP case, we would use the training set to find the “optimal” weights with the back-prop rule

Validation set: a set of examples used to tune the parameters of a classifier In the MLP case, we would use the validation set to find the “optimal” number of hidden units or determine a stopping point for the back-propagation algorithm

Test set: a set of examples used only to assess the performance of a fully-trained classifier In the MLP case, we would use the test to estimate the error rate after we have chosen the final model (MLP size and actual weights) After assessing the final model on the test set, YOU MUST NOT tune the model any further!

Why separate test and validation sets? The error rate estimate of the final model on validation data will be biased (smaller than the true error rate) since the validation set is used to select the final model After assessing the final model on the test set, YOU MUST NOT tune the model any further!

source : Introduction to Pattern Analysis,Ricardo Gutierrez-OsunaTexas A&M University, Texas A&M University

  • 37
    +1 for "YOU MUST NOT tune the model any further!" – stmax May 27 '14 at 9:51
  • 4
    What is the difference between "fit the parameters" and "tune the parameters"? – Metariat Aug 6 '15 at 16:21
  • 13
    @stmax Not to be pedantic, but once we have our final test error and we are NOT satisfied with the result, what do we do, if we cant tune our model any further?... I have often wondered about this case. – Spacey Oct 9 '16 at 1:50
  • 5
    @Tarantula you can continue tuning the model, but you'll have to collect a new test set. Of course no one does that ;) but violating that (especially when you repeat it several times) might lead to your model fitting the test set - which results in unrealistic / too optimistic scores. – stmax Oct 11 '16 at 7:39
  • 2
    I think this nomenclature is confusing. You are correct to say "YOU MUST NOT tune the model any further" after using the test set, but... what area you meant to do? Stop work on it? In reality you need a whole hierarchy of test sets. 1: Validation set - used for tuning a model, 2: Test set, used to evaluate a model and see if you should go back to the drawing board, 3: Super-test set, used on the final-final algorithm to see how good it is, 4: hyper-test set, used after researchers have been developing MNIST algorithms for 10 years to see how crazily overfit they are... etc. etc. – Timmmm Dec 23 '17 at 18:11

My 5 years experience in Computer Science taught me that nothing is better than simplicity.

The concept of 'Training/Cross-Validation/Test' Data Sets is as simple as this. When you have a large data set, it's recommended to split it into 3 parts:

++Training set (60% of the original data set): This is used to build up our prediction algorithm. Our algorithm tries to tune itself to the quirks of the training data sets. In this phase we usually create multiple algorithms in order to compare their performances during the Cross-Validation Phase.

++Cross-Validation set (20% of the original data set): This data set is used to compare the performances of the prediction algorithms that were created based on the training set. We choose the algorithm that has the best performance.

++Test set (20% of the original data set): Now we have chosen our preferred prediction algorithm but we don't know yet how it's going to perform on completely unseen real-world data. So, we apply our chosen prediction algorithm on our test set in order to see how it's going to perform so we can have an idea about our algorithm's performance on unseen data.

Notes:

-It's very important to keep in mind that skipping the test phase is not recommended, because the algorithm that performed well during the cross-validation phase doesn't really mean that it's truly the best one, because the algorithms are compared based on the cross-validation set and its quirks and noises...

-During the Test Phase, the purpose is to see how our final model is going to deal in the wild, so in case its performance is very poor we should repeat the whole process starting from the Training Phase.

  • 1
    it is easy and confusing to refer to the sets as phases and vice versa. – Matt O'Brien Mar 28 '15 at 20:51
  • @MattO'Brien Yeah you are right. It should have been better to use only one word. – innovIsmail Jun 3 '15 at 2:49
  • So I need to set aside the test set in the beginning to avoid contamination of data. Then from the remaining data I can run cross-validation multiple times, each time selecting training set and cross-validation set randomly. Would that be correct? – Fazzolini Dec 9 '16 at 7:47
  • @innovIsmail What if I skip the validation step? Say I have many algorithms and I trained them on the train set, then I just apply all of them to the test set, then I pick the one that has the best perform on the test set – KevinKim Mar 26 '17 at 4:42
  • 2
    It sounds to me like you're then just skipping the test step. – Mihai Danila May 5 '17 at 3:03

At each step that you are asked to make a decision (i.e. choose one option among several options), you must have an additional set/partition to gauge the accuracy of your choice so that you do not simply pick the most favorable result of randomness and mistake the tail-end of the distribution for the center 1. The left is the pessimist. The right is the optimist. The center is the pragmatist. Be the pragmatist.

enter image description here

Step 1) Training: Each type of algorithm has its own parameter options (the number of layers in a Neural Network, the number of trees in a Random Forest, etc). For each of your algorithms, you must pick one option. That’s why you have a training set.

Step 2) Validating: You now have a collection of algorithms. You must pick one algorithm. That’s why you have a test set. Most people pick the algorithm that performs best on the validation set (and that's ok). But, if you do not measure your top-performing algorithm’s error rate on the test set, and just go with its error rate on the validation set, then you have blindly mistaken the “best possible scenario” for the “most likely scenario.” That's a recipe for disaster.

Step 3) Testing: I suppose that if your algorithms did not have any parameters then you would not need a third step. In that case, your validation step would be your test step. Perhaps Matlab does not ask you for parameters or you have chosen not to use them and that is the source of your confusion.

1 It is often helpful to go into each step with the assumption (null hypothesis) that all options are the same (e.g. all parameters are the same or all algorithms are the same), hence my reference to the distribution.

2 This image is not my own. I have taken it from this site: http://www.teamten.com/lawrence/writings/bell-curve.png

  • 2
    I think the first sentence captures the fundamental answer to this question better than any of the other answers. "At each step that you are asked to make a decision (i.e. choose one option among several options), you must have an additional set/partition to gauge the accuracy of your choice..." – kobejohn Apr 6 '16 at 23:25
  • On question: If I want to find the best RandomForest (RF) and pretending there is only one hyper parameter of RF, which is the number of trees (N), then in step1, I run many RF with different N to build the forest; in step2, apply them on the validation test and pick the RF with N* that gives lowest error over validation test, then in step3, I apply RF with N* to the test set and get unbiased estimate of true test error of this RF with N*. But I could apply all my RFs on test set and pick the one with the lowest test error, which may not be N*. Then that is the point of doing validation step2? – KevinKim Mar 26 '17 at 4:53
  • 1
    @KevinKim : If you apply your test set to all RFs and use the results to make a further choice (pick another model), then you've just repeated the validation step. You have set your mind on "I need to get the lowest error with a model!". That is the point of training and validating, NOT testing. Testing is only about: I've trained and picked a model, now let's see how it performs "in general". Obviously the "general" test set is just another slice of data that may or may not be overfit, but the point is that YOU haven't knowingly overfit your model to it by choices. – Honeybear Mar 1 at 10:49
  • The three-wise split is just a very common approach (A) to give you an idea of how the model generalizes (B) with limited effort and (C) limited observed data. If you want to do better in terms of (B), you can do what you are suggesting: Use different validation sets to finetune for generalization. With limited data that is called cross-validation: Repeat the training and validation with varying training and test sets (for neural networks where training may take weeks this is not a thing). – Honeybear Mar 1 at 11:55
  • 1
    BUT: How the model will perform "in the real world" is still unknown. It is just a validated and tested assumption, that it will perform well on unseen data and for scientific purposes this is usually deemed enough. If you now go again and generate and pick models, until one fits validation set AND test set perfectly, then you degenerated your test set to a validation set. Better do cross-validation for that. In case your performance is constantly significantly worse on the test set, it always is an option that your data is just split badly and you'd want to restart with re-shuffled sets. – Honeybear Mar 2 at 9:53

It does not follow that you need to split the data in any way. The bootstrap can provide smaller mean squared error estimates of prediction accuracy using the whole sample for both developing and testing the model.

  • 1
    So you don't advocate cross-validation through splitting of large data-sets for predictive model testing / validation? – OFish Dec 15 '14 at 3:42
  • 8
    No, unless the dataset is huge or the signal:noise ratio is high. Cross-validation is not as precise as the bootstrap in my experience, and it does not use the whole sample size. In many cases you have to repeat cross-validation 50-100 times to achieve adequate precision. But in your datasets have > 20,000 subjects, simple approaches such as split-sample validation are often OK. – Frank Harrell Dec 15 '14 at 4:17
  • 1
    That's really good to know! Thanks. And coming from you, that's a great "source" of info. Cheers! – OFish Dec 15 '14 at 4:43
  • Could you provide a link to what you think is a good starting point for the bootstrapping method? – kobejohn Apr 6 '16 at 23:29
  • See Chapter 5 of my course notes at biostat.mc.vanderbilt.edu/rms – Frank Harrell Apr 6 '16 at 23:59

A typical machine learning task can be visualized as the following nested loop:

while (error in validation set > X) {
    tune hyper-parameters
    while (error in training set > Y) {
        tune parameters
    }
}

Typically the outer loop is performed by human, on the validation set, and the inner loop by machine, on the training set. You then need a 3rd test set to assess the final performance of the model.

In other words, validation set is the training set for human.

  • 1
    I like this explanation :) very concise – Honeybear Mar 1 at 11:54
  • very concise indeed. – Cong Hui Apr 20 at 5:56

Most supervised data mining algorithms follow these three steps:

  1. The training set is used to build the model. This contains a set of data that has preclassified target and predictor variables.
  2. Typically a hold-out dataset or test set is used to evaluate how well the model does with data outside the training set. The test set contains the preclassified results data but they are not used when the test set data is run through the model until the end, when the preclassified data are compared against the model results. The model is adjusted to minimize error on the test set.
  3. Another hold-out dataset or validation set is used to evaluate the adjusted model in step #2 where, again, the validation set data is run against the adjusted model and results compared to the unused preclassified data.

One way to think of these three sets is that two of them (training and validation) come from the past, whereas the test set comes from the "future". The model should be built and tuned using data from the "past" (training/validation data), but never test data which comes from the "future".

To give a practical example, let's say we are building a model to predict how well baseball players will do in the future. We will use data from 1899-2014 to create a test and validation set. Once the model is built and tuned on those data, we will use data from 2015 (actually in the past!) as a test set, which from the perspective of the model appears like "future" data and in no way influenced the model creation. (Obviously, in theory, we could wait for data from 2016 if we really want!)

Obviously I'm using quotes everywhere, because the actual temporal order of the data may not coincide with actual future (by definition all of the data generation probably took place in the actual past). In reality, the test set might simply be data from the same time period as the training/validation sets, that you "hold out". In this way, it had no influence on tuning the model, but those hold out data are not actually coming from the future.

  • 3
    After reading all the other answers, this answer made it "click" for me! You train with the train set, check that you're not overfitting with the validation set (and that the model and hyperparameters work with "unknown data"), and then you assess with the test set - "new data" - whether you now have any predictive powers..! – stolsvik Mar 15 '17 at 22:17

I would like to add to other very good answers here by pointing to a relatively new approach in machine learning called "differential privacy" (see papers by Dwork; the Win Vector Blog for more). The idea allows to actually reuse the testing set without compromising the final model performance. In a typical setting the test set is only used to estimate the final performance; ideally one is not even allowed to look at it.

As it is well described in this Win Vector blog (see other entries as well), it is possible to "use" the test set without biasing the model's performance. This is done using the special procedure called "differential privacy". The learner will not have direct access to the test set.

My Idea is that those option in neural network toolbox is for avoiding overfitting. In this situation the weights are specified for the training data only and don't show the global trend. By having a validation set, the iterations are adaptable to where decreases in the training data error cause decreases in validation data and increases in validation data error; along with decreases in training data error, this demonstrates the overfitting phenomenon.

Some people have confusion about why we use a validation set, so I will give a simple, intuitive explanation of what will happen if you don't use a validation dataset.

If you don't use a validation set, you will instead have to pick hyperparameters and decide when to stop training based on the performance of the model on the testing dataset. If you decide when to stop training based on the performance of the model on the testing dataset, you could just stop training when the model happens to do well on the testing dataset. Then when you report your results, you report the accuracy on the testing dataset. The problem with this is that you could say your model did really well when in fact it was just a random variation that caused it to do better on just the testing set.

If you use a validation set instead to decide when to stop training, the accuracy of the model on the testing set is more of an unbiased reflection of how well it performs on the task in general, and it shows that you didn't optimize the model just to perform well on the testing set.

protected by gung Feb 6 '17 at 15:39

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.