18
$\begingroup$

I am a bit confused about the application of cross-validation. So, if I have a big data set, I will split my data into test and training data and and perform validation on the test data. But if I have a small data set I would like to used cross-validation and then the validation is already performed within it.

What puzzles me is that lots of people split data, perform training on training data with cross-validation, and then perform validation on the test dataset. So they combine those two methods. Is this a proper way to do it? May I do only cross-validation since my data set is quite small?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ You can also do cross-validation to select the hyper-parameters of your model, then you validate the final model on an independent data set. The recommendation is usually to split the data in three parts, training, test and validation test sets. Use one for training the parameters of the model, one for model selection and finally one for validation of the final result. $\endgroup$
    – Tommy L
    Commented Apr 28, 2015 at 14:12
  • $\begingroup$ But since my data set is quite small i am able to do only croosvalidation what i could use for looking at the roc curve. Is it right? $\endgroup$
    – Anni
    Commented Apr 28, 2015 at 14:45
  • $\begingroup$ Note that @TommyL's splitting in 3 parts does not say anything how exactly the splitting is performed. On both levels you can do resampling or an independent test set. $\endgroup$
    – cbeleites
    Commented Apr 29, 2015 at 8:58

1 Answer 1

Reset to default
27
$\begingroup$

Let's look at three different approaches

  1. In the simplest scenario one would collect one dataset and train your model via cross-validation to create your best model. Then you would collect another completely independent dataset and test your model. However, this scenario is not possible for many researchers given time or cost limitations.

  2. If you have a sufficiently large dataset, you would want to take a split of your data and leave it to the side (completely untouched by the training). This is to simulate it as a completely independent dataset set even though it comes from the same dataset but the model training won't take any information from those samples. You would then build your model on the remaining training samples and then test on these left-out samples.

  3. If you have a smaller dataset, you may not be able to afford to simply ignore a chunk of your data for model building. As such, the validation is performed on every fold (k-fold CV?) and your validation metric would be aggregated across each validation.

To more directly answer your question, yes you can just do cross-validation on your full dataset. You can then use your predicted and actual classes to evaluate your models performance by whatever metric you prefer (Accuracy, AUC, etc.)

That said, you still probably want to look in to repeated cross-validation to evaluate the stability of your model. Some good answers regarding this are here on internal vs. external CV and here on the # of repeats

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I agree with this. Also, I would like to add that it is generally problematic to perform model building and validation on the same data set. Your p-values will be too low, for instance. Essentially you have overfit to the data. However, if it is impossible to obtain more samples, using resampling, such as CV, may be your only choice. In this case, you might also benefit from bootstrapping. $\endgroup$
    – Tommy L
    Commented Apr 29, 2015 at 6:31
  • $\begingroup$ I have a sufficiently large dataset, and I'd like to use approach (2). Suppose the dataset is split into training set A and independent set B. My question is when I fit a model and perform k-fold validation, this is happening entirely on A right? (but what's the purpose of k-fold validation here then?) $\endgroup$
    – masfenix
    Commented Jul 25, 2015 at 22:05
  • $\begingroup$ @masfenix The k-fold is run entirely on set A. A is split in to the folds and you tune the model on those sets and finally evaluate the model on set B. If you require further details I suggest opening a question of your own. $\endgroup$
    – cdeterman
    Commented Jul 27, 2015 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.