If I have a data set of 500 observations, should i divide it in a train and test set with for example 375 (75%) train observations and 125 (25%) test observations, and perform cross-validation on the train set?
Yes, you should do it as the initial step, regardless of the situation (
test_size=0.2 is also a reasonable default in
Or should I perform the cross-validation on the entire data set?
Try to avoid it as much as possible and to stick to the traditional guidelines, but what most guides and books don't tell you is that YOU MAY HAVE TO.
There is theory, and then there is practice... If the labeled data set is small, if there is no practical way to increase it and there is a high risk of the training set being non-representative to a large extent, you may have to give up a separate test set.
Before you launch into non-standard analytic pathways, however, you need to ensure that you understand what the risks associated with them are and the impact on your conclusions. Ultimately, the decision is a pragmatic one based on the needs and the risks associated with the project. In regulated fields, such as finance or healthcare, you need to be cautious, but if all the outcome is whether to colour a 'Buy' button green or blue then the risks are lower.
What you will lose by following this non-standard analytic pathway:
- you will be giving up the second round of "generalization testing" (with CV on the training set for model selection/tuning being the first one)
- you will be exposing yourself to the risk of overfitting your data by having no hold out to check whether the model is stable or not when applied to new data
- you will be performing more hypothesis tests, leading to a higher risk of making a false discovery (related to point 2)
- you will exploit more researcher degrees of freedom (garden of forking paths) - another route to overfitting
What you will gain:
- you will be gaining the confidence that your model is trained on the more representative samples
- you will have a deeper insight into your dataset, hopefully including biases, errors, and the desired data generating processes
- you will be better placed to recommend improved plans, should any opportunity arise to redo the whole thing
So, with the understanding of the above risks and benefits, how does one make a decision on giving up a separate test set?
Here's a suggested workflow:
1) After selecting and tuning an algorithm using the standard method (training CV + fit on the entire training set + testing on the separate test set), go back to the train/test split, split the data set differently a few times (e.g. using a different
random_state parameter value in scikit-learn), each time re-executing the "training CV + train set fit + testing" cycle and observing the scores.
2) If the difference between the training CV scores and the testing score is relatively consistent - great! No need to do anything else.
3) If, however, the difference between the mean training CV score and the testing score is inconsistent, and especially if the fact which one is larger varies between these random data set splitting iterations - YOU'VE GOT A SAMPLING PROBLEM.
a) Find another algorithm that will generalize well to the testing set regardless of the train/test split (this may be very challenging, given that you're probably working with a small labeled data set to begin with).
b) Identify and eliminate the splitting bias systematically (this may be not practical either).
c) Keep acquiring more representative labeled data until this problem goes away.
d) Declare this being a "failed analysis" and postpone/cancel model launch, since getting rid of the separate round of generalization testing has high analytic risks associated with it.
e) Only if all else fails, give up a separate test data set (i.e. cut out the separate generalization testing step from the process) and select the model via the cross-validation on the entire labeled data set.
Observe not just the mean CV test score (which is the mean of the fold means) when making model selection/tuning decisions, but also min/max/std of the mean test scores for the individuals folds.