2
$\begingroup$

I'm writing code to perform classification on novel data sets in our lab, and I'm confusing myself as to when I should be performing cross-validation. From this question I understand that I should choose hyper-parameters by cross-validation.

However, in the case that there are no hyper-parameters to tune, and all I want is to return labels for data points, do I need to cross-validate?

$\endgroup$
4
  • $\begingroup$ Check stats.stackexchange.com/questions/1826/… for start. $\endgroup$
    – Tim
    Jul 12, 2016 at 15:05
  • $\begingroup$ Thanks. From that I understand that I don't need to cross-validate if there are no parameters to tune. Is that right? $\endgroup$ Jul 12, 2016 at 15:22
  • $\begingroup$ What type of classification scheme are you using that has no hyper-parameters to tune? Is it one that might nevertheless be prone to over-fitting? You might then find need some type of validation, although bootstrapping might be more useful than cross-validation. Numbers of cases, predictors, and class representation (particularly the number of cases in the smaller classes) would help here. $\endgroup$
    – EdM
    Jul 12, 2016 at 15:32
  • $\begingroup$ I'm using a Naive Bayes on 2 classes and 10 features, with about 20,000 positive cases and 500 negative. The question about what to do if there are no hyperparameters is hypothetical, since I am performing cross-validation for hyperparameters like data whitening and class balancing. However I'm confused if I need to do any kind of cross-validation after that, or whether I can just classify. I don't think I'm interested in e.g. estimating the model performance - I just want labels for each data point. $\endgroup$ Jul 12, 2016 at 15:41

1 Answer 1

3
$\begingroup$

Ideally, always cross-validate because getting the results on different splits allows you to better estimate your model performance. In practice, you do it if you estimate that you don't have enough data to train/test your model properly.

This is not directly linked to hyper parameters. e.g. a valid experimental protocol is to split your data in training/validation and test sets. You train your model on the training set You choose your hyper-parameters that perform the best on the validation set. When all done, you evaluate your final model on the test set.

$\endgroup$
3
  • $\begingroup$ Thanks. If I'm not interested in estimating model performance, and I just want to attach classes to data points, could I skip cross-validation? Or is it good practice to always cross-validate? $\endgroup$ Jul 12, 2016 at 15:46
  • $\begingroup$ If you just want to attach labels and don't care how accurate those labels are, why even build a model? Just assign labels arbitrarily. If you care how good those labels are then use something like cross validation or an independent test set to measure performance. $\endgroup$
    – dsaxton
    Jul 12, 2016 at 23:00
  • $\begingroup$ Yes, cross-validation is only for evaluation. When in production, you want to use all your training data to learn the best possible model. $\endgroup$
    – paul_gay
    Mar 1, 2018 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.