# What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab.
It divided the raw data set into three parts:

1. training set
2. validation set
3. test set

I notice in many training or learning algorithm, the data is often divided into 2 parts, the training set and the test set.

My questions are:

1. what is the difference between validation set and test set?
2. Is the validation set really specific to neural network? Or it is optional.
3. To go further, is there a difference between validation and testing in context of machine learning?
-
The question is answered in the book Elements of statistical learning page 222. The validation set is used for model selection, the test set for final model (the model which was selected by selection process) prediction error. – mpiktas Nov 28 '11 at 11:47
@mpiktas Are you referring to the chapter "Model Assessment and Selection"? – Celdor Jun 1 '15 at 7:16
Yes. The page number was from 5th print edition. – mpiktas Jun 1 '15 at 7:20
You might want to also see: stats.stackexchange.com/questions/9357/…, where the question was "Why not more than three?" – Wayne Sep 29 '15 at 14:28

Normally to perform supervised learning you need two types of data sets:

1. In one dataset (your "gold standard") you have the input data together with correct/expected output, This dataset is usually duly prepared either by humans or by collecting some data in semi-automated way. But it is important that you have the expected output for every data row here, because you need for supervised learning.

2. The data you are going to apply your model to. In many cases this is the data where you are interested for the output of your model and thus you don't have any "expected" output here yet.

While performing machine learning you do the following:

1. Training phase: you present your data from your "gold standard" and train your model, by pairing the input with expected output.
2. Validation/Test phase: in order to estimate how well your model has been trained (that is dependent upon the size of your data, the value you would like to predict, input etc) and to estimate model properties (mean error for numeric predictors, classification errors for classifiers, recall and precision for IR-models etc.)
3. Application phase: now you apply your freshly-developed model to the real-world data and get the results. Since you normally don't have any reference value in this type of data (otherwise, why would you need your model?), you can only speculate about the quality of your model output using the results of your validation phase.

The validation phase is often split into two parts:

1. In the first part you just look at your models and select the best performing approach using the validation data (=validation)
2. Then you estimate the accuracy of the selected approach (=test).

Hence the separation to 50/25/25.

In case if you don't need to choose an appropriate model from several rivaling approaches, you can just re-partition your set that you basically have only training set and test set, without performing the validation of your trained model. I personally partition them 70/30 then.

-
Why wouldn't I choose the best performing model based on the test set, getting rid of the validation set altogether? – Sebastian Graf Nov 9 '14 at 14:31
Is it because of overfitting? Or because we want some independent statistics based on the test result, just for error estimation? – Sebastian Graf Nov 9 '14 at 14:42
@Sebastian [If you only use the test set: ]"The test set error of the final chose model will underestimate the true test error, sometimes significantly" [Hastie et al] – user695652 Jun 2 '15 at 20:09
The validation set is often used to tune hyper-parameters. For example, in the deep learning community, tuning the network layer size, hidden unit number, regularization term(wether L1 or L2) depends on the validation set – xiaohan2012 Oct 13 '15 at 10:52
What is the correct way to split the sets? Should the selection be random? What if you have pictures that are similar? Won't this damage your ability to generalize? If you have two sets taken in separate locations wouldn't it be better to take one as training set and the other as the test set? – Yonatan Simson Feb 3 at 10:36

Training set: a set of examples used for learning: to fit the parameters of the classifier In the MLP case, we would use the training set to find the “optimal” weights with the back-prop rule

Validation set: a set of examples used to tune the parameters of a classifier In the MLP case, we would use the validation set to find the “optimal” number of hidden units or determine a stopping point for the back-propagation algorithm

Test set: a set of examples used only to assess the performance of a fully-trained classifier In the MLP case, we would use the test to estimate the error rate after we have chosen the final model (MLP size and actual weights) After assessing the final model on the test set, YOU MUST NOT tune the model any further!

Why separate test and validation sets? The error rate estimate of the final model on validation data will be biased (smaller than the true error rate) since the validation set is used to select the final model After assessing the final model on the test set, YOU MUST NOT tune the model any further!

source : Introduction to Pattern Analysis,Ricardo Gutierrez-OsunaTexas A&M University, Texas A&M University

-
+1 for "YOU MUST NOT tune the model any further!" – stmax May 27 '14 at 9:51
What is the difference between "fit the parameters" and "tune the parameters"? – Matemattica Aug 6 '15 at 16:21
Very clear explanation: +1 – VolAnd Oct 5 '15 at 13:52

At each step that you are asked to make a decision (i.e. choose one option among several options), you must have an additional set/partition to gauge the accuracy of your choice so that you do not simply pick the most favorable result of randomness and mistake the tail-end of the distribution for the center 1. The left is the pessimist. The right is the optimist. The center is the pragmatist. Be the pragmatist.

Step 1) Training: Each type of algorithm has its own parameter options (the number of layers in a Neural Network, the number of trees in a Random Forest, etc). For each of your algorithms, you must pick one option. That’s why you have a training set.

Step 2) Validating: You now have a collection of algorithms. You must pick one algorithm. That’s why you have a test set. Most people pick the algorithm that performs best on the validation set (and that's ok). But, if you do not measure your top-performing algorithm’s error rate on the test set, and just go with its error rate on the validation set, then you have blindly mistaken the “best possible scenario” for the “most likely scenario.” That's a recipe for disaster.

Step 3) Testing: I suppose that if your algorithms did not have any parameters then you would not need a third step. In that case, your validation step would be your test step. Perhaps Matlab does not ask you for parameters or you have chosen not to use them and that is the source of your confusion.

1 It is often helpful to go into each step with the assumption (null hypothesis) that all options are the same (e.g. all parameters are the same or all algorithms are the same), hence my reference to the distribution.

2 This image is not my own. I have taken it from this site: http://www.teamten.com/lawrence/writings/bell-curve.png

-
I think the first sentence captures the fundamental answer to this question better than any of the other answers. "At each step that you are asked to make a decision (i.e. choose one option among several options), you must have an additional set/partition to gauge the accuracy of your choice..." – kobejohn Apr 6 at 23:25

My 5 years experience in Computer Science taught me that nothing is better than simplicity.

The concept of 'Training/Cross-Validation/Test' Data Sets is as simple as this. When you have a large data set, it's recommended to split it into 3 parts:

++Training set (60% of the original data set): This is used to build up our prediction algorithm. Our algorithm tries to tune itself to the quirks of the training data sets. In this phase we usually create multiple algorithms in order to compare their performances during the Cross-Validation Phase.

++Cross-Validation set (20% of the original data set): This data set is used to compare the performances of the prediction algorithms that were created based on the training set. We choose the algorithm that has the best performance.

++Test set (20% of the original data set): Now we have chosen our preferred prediction algorithm but we don't know yet how it's going to perform on completely unseen real-world data. So, we apply our chosen prediction algorithm on our test set in order to see how it's going to perform so we can have an idea about our algorithm's performance on unseen data.

Notes:

-It's very important to keep in mind that skipping the test phase is not recommended, because the algorithm that performed well during the cross-validation phase doesn't really mean that it's truly the best one, because the algorithms are compared based on the cross-validation set and its quirks and noises...

-During the Test Phase, the purpose is to see how our final model is going to deal in the wild, so in case its performance is very poor we should repeat the whole process starting from the Training Phase.

-
it is easy and confusing to refer to the sets as phases and vice versa. – Matt O'Brien Mar 28 '15 at 20:51
@MattO'Brien Yeah you are right. It should have been better to use only one word. – innovIsmail Jun 3 '15 at 2:49

It does not follow that you need to split the data in any way. The bootstrap can provide smaller mean squared error estimates of prediction accuracy using the whole sample for both developing and testing the model.

-
So you don't advocate cross-validation through splitting of large data-sets for predictive model testing / validation? – OFish Dec 15 '14 at 3:42
No, unless the dataset is huge or the signal:noise ratio is high. Cross-validation is not as precise as the bootstrap in my experience, and it does not use the whole sample size. In many cases you have to repeat cross-validation 50-100 times to achieve adequate precision. But in your datasets have > 20,000 subjects, simple approaches such as split-sample validation are often OK. – Frank Harrell Dec 15 '14 at 4:17
That's really good to know! Thanks. And coming from you, that's a great "source" of info. Cheers! – OFish Dec 15 '14 at 4:43
Could you provide a link to what you think is a good starting point for the bootstrapping method? – kobejohn Apr 6 at 23:29
See Chapter 5 of my course notes at biostat.mc.vanderbilt.edu/rms – Frank Harrell Apr 6 at 23:59

Most supervised data mining algorithms follow these three steps:

1. The training set is used to build the model. This contains a set of data that has preclassified target and predictor variables.
2. Typically a hold-out dataset or test set is used to evaluate how well the model does with data outside the training set. The test set contains the preclassified results data but they are not used when the test set data is run through the model until the end, when the preclassified data are compared against the model results. The model is adjusted to minimize error on the test set.
3. Another hold-out dataset or validation set is used to evaluate the adjusted model in step #2 where, again, the validation set data is run against the adjusted model and results compared to the unused preclassified data.
-

I would like to add to other very good answers here by pointing to a relatively new approach in machine learning called "differential privacy" (see papers by Dwork; the Win Vector Blog for more). The idea allows to actually reuse the testing set without compromising the final model performance. In a typical setting the test set is only used to estimate the final performance; ideally one is not even allowed to look at it.

As it is well described in this Win Vector blog (see other entries as well), it is possible to "use" the test set without biasing the model's performance. This is done using the special procedure called "differential privacy". The learner will not have direct access to the test set.

-

My Idea is that those option in neural network toolbox is for avoiding overfitting. In this situation the weights are specified for the training data only and don't show the global trend. By having a validation set, the iterations are adaptable to where decreases in the training data error cause decreases in validation data and increases in validation data error; along with decreases in training data error, this demonstrates the overfitting phenomenon.

-

One way to think of these three sets is that two of them (training and validation) come from the past, whereas the test set comes from the "future". The model should be built and tuned using data from the "past" (training/validation data), but never test data which comes from the "future".

To give a practical example, let's say we are building a model to predict how well baseball players will do in the future. We will use data from 1899-2014 to create a test and validation set. Once the model is built and tuned on those data, we will use data from 2015 (actually in the past!) as a test set, which from the perspective of the model appears like "future" data and in no way influenced the model creation. (Obviously, in theory, we could wait for data from 2016 if we really want!)

Obviously I'm using quotes everywhere, because the actual temporal order of the data may not coincide with actual future (by definition all of the data generation probably took place in the actual past). In reality, the test set might simply be data from the same time period as the training/validation sets, that you "hold out". In this way, it had no influence on tuning the model, but those hold out data are not actually coming from the future.

-