2
$\begingroup$

If you cannot acquire a workable test data-set to test a model, you can use cross validation as an alternative to validate the model.

However, I'm unsure of the true end purpose of cross validation...

Is CV useful for simply generating a performance metric by providing a way to test your model? Or is it a means to creating a final model?

Put differently: do I use my "best" model from whichever given fold of the CV process (i.e., the "best" model using some partition of the data) as my final model? Or do I perform CV, report my performance metric, and then re-create the model using ALL of my available data to create a final model?

$\endgroup$

2 Answers 2

2
$\begingroup$

Is CV useful for simply generating a performance metric by providing a way to test your model? Or is it a means to creating a final model?

For both. It generates performance metrics to test your model (perhaps, on different parameters).

It is also a means of creating the final model, because you would "often" use cross-validation to choose the best parameters of the model. You then use the best parameters on all of the data to generate the final model. However, while reporting please report the cross-validation error.

$\endgroup$
5
  • $\begingroup$ What exactly is "cross-validation error"? How is it calculated? $\endgroup$ Oct 8, 2016 at 4:30
  • 1
    $\begingroup$ You train a model on training data. Use that model to predict on the test data. The performance of the model on the test data is called cross-validation error. $\endgroup$
    – discipulus
    Oct 8, 2016 at 4:31
  • $\begingroup$ Ah. So it's just relative to what it's reporting performance of (i.e., the model's ability to predict values for the test data). It could theoretically consist of MSE, RMSE, AUC, MAE, etc. , then? $\endgroup$ Oct 8, 2016 at 4:34
  • 1
    $\begingroup$ Yes! Metric is your choice and depends on the task at hand and also on the data. Cross-validation error is used more generically; it may be accuracy or AUC or accuracy or similar in clasification and MAE or MSE or similar in regression. $\endgroup$
    – discipulus
    Oct 8, 2016 at 4:37
  • $\begingroup$ Error and accuracy are not the same; they are the flip sides of the same coin. $\endgroup$
    – rolando2
    Jan 28, 2017 at 23:28
0
$\begingroup$

From Zhang 1997:

The purpose of data splitting is to choose the best model or to study model stability and predictive performance. The final estimation of the model parameters should come from the entire data set (Myers, 1990; Shao, 1993).

Myers RH. 1990. Classical and modern regression with applications. 2nd Edn. Boston: PWS-KENT Publishing Company.

Shao J. 1993. Linear model selection by cross-validation. Journal of American Statistical Association 88: 486±494.

ZHANG, LIANJUN. 1997. Cross-validation of non-linear growth functions for modelling tree height–diameter relationships. Annals of Botany 79.3: 251-257.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.