1
$\begingroup$

So as far as i know, there are many ways to evaluate a M.L/D.L model. Some of them, for the sake of the example, are k-fold cross validation and test/validation/train splits.

Now with regards to validation sets. Say if you have a dataset of 100 rows and 3 columns, 2 of which [columns] are independent features & the last one [3rd column] is the dependent variable vector and wanted to split dataset into training, validation and test sets, would the validation set be a part of the training set? [seperate from the test set?]

In other words, it is most common, with small datasets, to use the 80/20 split; where 80% of the data goes to training and 20% goes to testing. My question, is the validation set within the 80% such that:

(validation + training set) + test set = 100% of the data?

i.e. (20% + 60%) + 20% = 100%

Or am I getting this all mixed up and the validation set is in fact the test set but for a model that is still having its parameters tunes?

P.S I apologize for what seems to be a mess of a question, but this in fact reflects the state in which my mind seems to be in with this topic.

$\endgroup$
1
$\begingroup$

It's arbitrary.

The use of cross-validation for testing the model is not recommended, because then you use the same data for developing the model and testing it. Although cross-validation is fine for... validating, i.e. picking a model among several contenders. So let's assume you want to test on a brand new, unseen test set.

The use of 80/20 proportions is just a rule of thumb. If you have lots of data, using 20% of the data for testing can be a waste and with only 1% of the data, you could approach the true generalisation error with very little variance already (variance over sampling of the test set). Conversely, if you are doing classification over more than 20 classes, obviously using only 20 test datapoints will not suffice.

Even if you give us your number of datapoints in your question, without more information, I don't see how anybody could tell you what split sizes to use (which is what your question amounts to). The dataset could be very easy to solve even with a couple of datapoints (for example, if you learn the behavior of logic gates without any noise...).

Since the question is a bit fuzzy, let me remark that:

  • The rule might work because when people gather datasets, they try to make it at least large enough so that the rule work?
  • Maybe this rule was more relevant when datasets were small (and you are in this case), but I don't know the theory behind these numbers.

In general, however, you should always consider the validation data as part of the training data; data that you can use during the process of developing your model. If 20 datapoints for the test set seems to be enough for your purpose (comparing models, or estimating the true generalisation error), you will take the validation set from the 80 remaining points. Basically, choose the test set size first, and then choose the validation set size by looking at what's left, and draw this amount from the rest of the data.

By the way, if you are doing classification, you could use stratified sampling to sample your test set and estimate the generalisation error better.

$\endgroup$
4
  • $\begingroup$ I think you may have misunderstood the nature of my question, which is fair. It's not that i am trying to find out the best split. I'm trying to figure out whether in any given situation, i should have either a (validation set + test set + training set) or just a (training set + validation set) $\endgroup$ – Kamal Raydan Oct 1 '20 at 15:57
  • $\begingroup$ Because when you mentioned to choose your test data first, and then the rest of points would fall under validation, what happened to the training points? $\endgroup$ – Kamal Raydan Oct 1 '20 at 15:57
  • $\begingroup$ What I was saying is: 1: pick your test set. 2. pick your validation set (if necessary, i.e. if you need to choose hyperparameters or want to do early stopping, what you can call model selection in general). The rest of the data (not in the test set, not in the validation set) is your train set. $\endgroup$ – bomzh Oct 1 '20 at 21:20
  • $\begingroup$ You need to be clear as to why people validate and test. Indeed, that's not really what my question answer: I try to tackle the more specific question of how to choose the split sizes. $\endgroup$ – bomzh Oct 1 '20 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.