3
$\begingroup$

I am trying to get my head straight on terminology which appears confusing. I know there are three 'splits' of data used in Machine learning models.:

  1. Training Data - Train the model
  2. Validation Data - Cross validation for model selection
  3. Testing Data - Test the generalisation error.

Now, as far as I am aware, the validation data is not always used as one can use k-fold cross-validation, reducing the need to further reduce ones dataset. The results of which are known as the validation accuracy. Then once the best model is selected, the model is tested on a 33% split from the initial data set (which has not been used to train). The results of this would be the testing accuracy?

Is this the right way around? or is vice versa? I am finding conflicting terminology used online! I am trying to find some explanations why my validation error is larger than my testing error, but before I find a solution, i would like to get my terminology correct.

Thanks.

$\endgroup$
  • $\begingroup$ Please also take a look at my answer on a similar post which explains the key differences. Specially the last part on validation set. $\endgroup$ – Esmailian Apr 9 at 23:07
2
$\begingroup$

There isn't a standard terminology in this context (and I have seen long discussions and debates regarding this topic), so I completely understand you, but you should get used to different terminology (and assume that terminology might not be consistent or it change across sources).

I would like to point out a few things:

  • I have never seen people use the expression "validation accuracy" (or dataset) to refer to the test accuracy (or dataset), but I have seen people use the term "test accuracy" (or dataset) to refer to the validation accuracy (or dataset). In other words, the test (or testing) accuracy often refers to the validation accuracy, that is, the accuracy you calculate on the data set you do not use for training, but you use (during the training process) for validating (or "testing") the generalisation ability of your model or for "early stopping".

  • In k-fold cross-validation, people usually only mention two datasets: training and testing (or validation).

  • k-fold cross-validation is just a way of validating the model on different subsets of the data. This can be done for several reasons. For example, you have a small amount of data, so your validation (and training) dataset is quite small, so you want to have a better understanding of the model's generalisation ability by validating it on several subsets of the whole dataset.

  • You should likely have a separate (from the validation dataset) dataset for testing, because the validation dataset can be used for early stopping, so, in a certain way, it is dependent on the training process

I would suggest to use the following terminology

  • Training dataset: the data used to fit the model.
  • Validation dataset: the data used to validate the generalisation ability of the model or for early stopping, during the training process.
  • Testing dataset: the data used to for other purposes other than training and validating.

Note that some of these datasets might overlap, but this might almost never be a good thing (if you have enough data).

$\endgroup$
  • $\begingroup$ If the testing dataset overlaps with either of the others, it is definitely not a good thing. The test accuracy must measure performance on unseen data. If any part of training saw the data, then it isn't test data, and representing it as such is dishonest. Allowing the validation set to overlap with the training set isn't dishonest, but it probably won't accomplish its task as well. (e.g., if you're doing early stopping, and your validation set and training sets overlap, overfitting may occur and not be detected.) $\endgroup$ – Ray Apr 7 at 23:44
  • $\begingroup$ @Ray I didn't say it is a good thing. Indeed, see my point "You should likely have a separate (from the validation dataset) dataset for testing...". $\endgroup$ – nbro Apr 7 at 23:46
  • $\begingroup$ You said "If that's a 'good' thing or not, it's another question." I suspected from the rest that you understood the problems that that overlap would cause, but the problems with that should be made very clear, since contaminating your test data with training samples completely ruins its value. $\endgroup$ – Ray Apr 7 at 23:48
  • $\begingroup$ @Ray I wanted more to refer to the overlap between the training and validation datasets. Anyway, I think it's good that you wanted to clarify or emphasise this point. I edited my answer to emphasise this point. $\endgroup$ – nbro Apr 7 at 23:51
1
$\begingroup$

@nbro's answer is complete. I just add a couple of explanations to supplement. In more traditional textbooks data is often partitioned into two sets: training and test. In recent years, with more complex models and increasing need for model selection, development sets or validations sets are also considered. Devel/validation should have no overlap with the test set or the reporting accuracy/ error evaluation is not valid. In the modern setting: the model is trained on the training set, tested on the validation set to see if it is a good fit, possibly model is tweaked and trained again and validated again for multiple times. When the final model is selected, the testing set is used to calculate accuracy, error reports. The important thing is that the test set is only touched once.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.