The question might be not so clear. However, my confusion comes from the following approach: Suppose I use training, testing and validation sets. First, I split my data into training and testing. To tune my parameters (e.g., for pruning a classification tree), I then split my training set into validation and XX. I cannot use the term training set again but how is this set (training-valiation) called?
$\begingroup$
$\endgroup$
asked Dec 29, 2015 at 11:09
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
I agree that the employed terminology is not so great. Here is one way to make it unambiguous, which is used in the AMI Corpus documentation:
First, you can split the data into seen and unseen data. You then split your seen data into validation set and training set. The unseen test is also called test set. The validation set is also called development set, or hold-out data.
answered Jan 3, 2016 at 0:12
-
1$\begingroup$ There is even more ambiguity because in my field validation would correspond to the test set rather than the development set (or even to data obtained by a validation study, not by splitting some data set). Not knowing the AMI definitions, I've used "optimization set" or "optimization test set" for the inner "validation set". $\endgroup$ Commented Jan 5, 2016 at 17:21
-
$\begingroup$ @cbeleites Interesting. What is your field? is development set synonymous with validation set as well? $\endgroup$ Commented Jan 5, 2016 at 17:25
-
$\begingroup$ I'm analytical chemist / chemometrician. I've not met the term "development set" before. At some point I decided always to spell out which set I refer to by which term. $\endgroup$ Commented Jan 5, 2016 at 18:07
-
$\begingroup$ @cbeleites When even the terms to split the data are ambiguous, one may wonder if stats/ML terminology doesn't need some serious cleaning :/ $\endgroup$ Commented Jan 7, 2016 at 18:18
$\begingroup$
$\endgroup$
1
You shouldn't split your training set, because initially you split your data to training set, cross-validation set and test set. Usually corresponding ratios are 60% + 20% + 20%
-
$\begingroup$ Thank you for the quick answer. I see your point. However, since I'd like to run a cross-validation to get more robust results, I guess I cannot just split it in the beginning. Of course, if I would just to a one-time random split, your solution would clearly be correct. At least this is how I think about it. But I see, I was not clear about this! $\endgroup$ Commented Dec 29, 2015 at 12:56