I'm surprised that this question has not been asked before (Maybe I just couldn't find it), but from where should the validation set come from?

Should we split the the total dataset into training, validation and testing datasets?

Or should we first split the total dataset into training and testing sets, then split the training set or the testing set into training and validation subsets?

Edit: Doesn't each subset of the data have its own statistical characteristics? Variance, mean, bias .. etc. So the question is for ML in production, should the validation set come from the training distribution or the test.

More clarification to the down voters ********, It's possible that in production, the testing and training set come from two distributions, then which one to draw the validation set from?

  • 1
    $\begingroup$ What difference does it make between the two described splits? $\endgroup$ – Jenkar Aug 17 '17 at 13:09
  • $\begingroup$ I don't really see any meaningful difference between those two options. Perhaps you are interested in the proportion of the data that should be in each dataset? $\endgroup$ – mkt - Reinstate Monica Aug 17 '17 at 13:10
  • $\begingroup$ Nope. The proportion of the data is not what I'm asking about. Check the edits $\endgroup$ – MAA Aug 17 '17 at 13:23
  • $\begingroup$ What's "********" supposed to mean? $\endgroup$ – Firebug Aug 17 '17 at 16:11
  • $\begingroup$ I've come across modeling scenarios where you have a limited dataset that you can use for testing from the current year, and a larger dataset from last year. In that case, you are wondering if your model trained on last years data will be able to extrapolate to this year. In that case, I'd say split the current year into validation + test sets. Though I could see a case for splitting last years data into train + validation sets. $\endgroup$ – Azmisov Aug 3 at 22:25

Say you have 9 apples. What difference does it make if you:

  • put 3 in the basket A, put the next 3 in basket B, and put the remaining 3 in the basket C,
  • put 3 in the basket A, then take remaining 6 and from those apples put 3 in basket B and 3 in basket C,
  • take 6 and from those apples put 3 in basket A and 3 in basket B, then put the remaining 3 in basket C.

It makes no difference.

| cite | improve this answer | |

So let me reiterate what are different sets for first. Training set is to let your algorithm learn its parameters so that it can minimize its loss function based on all the data in the training set.

Validation set is to tune the hyperparameter of the algorithm(there are hyperparameters for most of ML algorithm implementations), it does not do this inside the training process, instead the trained model is used to predict on the validation data to evaluate how good this trained model is. And you variate your hyperparameters in this step so you can find what is the best set of hyperparameters to optimize the model. Therefore your best model kind of "knows" your validation data even it's not directly trained on it. For most of the cases, we are interested in a model that that gives the minimum Expected prediction error, which is the expectation of prediction error given all possible training data sets, however in reality we can only approach this value so that's why we use n-fold cross-validation.

Now test set is to assess the performance of your final trained model, your trained model with best-tuned hyper-parameter is totally oblivious of this data set, so the "test error" can serve to check how good your model can be generalized.

One confusion I encounter is sometimes people mix up validation error and test set error (tho it's not considered to be errorneous to use it interchangeably). I think in one instance validation error can be referred as test error which is the model is sparse therefore validation set is served as test set. Otherwise it's a bit confusing.

I personally do not think it's totally necessary to have a test set, as I read in ESL that n-fold cross validation can be unbiased to the expected prediction error and you can just check error variance to see how stable your model is anyway, so I guess if you have a lot of data, it does not hurt to have a test set.

| cite | improve this answer | |

Based on basics of data science taught during my masters and my understanding of working in the finance modeling:

First you collect your data for a specific vintage, say 2017 data. You divide this 70:30 or 80:20 into two sets: Train and Test 1. Train set - on which model is trained (maybe you're comparing models Random forest model performs better than Logistic) 2. Test set - on which model is selected (you see how the model performs on test set. You realize RF model is overfitted, i.e. Performance in test drops much more than Train, whereas Logistic is robust. You perform checks to make RF model more stable by going back to train and making finetuning parameters. or maybe You decide to go with logistic even though RF performed better on Train) i.e. Your model will be finally selected with the help of Test. 3. Model Validation - Final check, nothing should be used from this to change in the training. This could be a stability test for ex. Vintage 2018. This is called out-of-time validation. If this cannot be procured and you know it before hand, try to maybe remove two month of data e.g. Feb&August (this should be done carefully since certain months have different trends like December) as Validation and split rest into train: test 70:30. Else you can split the whole data randomly into 60:25:15 or any other combination. Usually the issue is loss of information when all three are coming from same dataset, so try to have enough observations.

| cite | improve this answer | |

Nuts and bolts of applying deep learning

Never mind. Andrew Ng states here that the validation set and the testing set should come from the same distribution (42:10).

But what does he know?

"Splitting your data. In most deep learning problems, train and test come from different distributions. For example, suppose you are working on implementing an AI powered rearview mirror and have gathered 2 chunks of data: the first, larger chunk comes from many places (could be partly bought, and partly crowdsourced) and the second, much smaller chunk is actual car data.

In this case, splitting the data into train/dev/test can be tricky. One might be tempted to carve the dev set out of the training chunk like in the first example of the diagram below. (Note that the chunk on the left corresponds to data mined from the first distribution and the one on the right to the one from the second distribution.)

This is bad because we usually want our dev and test to come from the same distribution. The reason for this is that because a part of the team will be spending a lot of time tuning the model to work well on the dev set, if the test set were to turn out very different from the dev set, then pretty much all the work would have been wasted effort."


| cite | improve this answer | |
  • 2
    $\begingroup$ ... The training set also should come from the same distribution. If not, the data used to train will (most likely) not reflect features that you want your method to learn, or at least, it will miss out on important features. This is the same reason that the validation & testing set should come from the same distribution. $\endgroup$ – Jenkar Aug 17 '17 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.