2
$\begingroup$

I have some doubts about cross validation and scoring a new set of observations.

Let's say I want to predict $y=b_0 + b_1x_1$ and have built a 10-fold cross-validation data set, run a regression model and calculated RMSE, MAE, CV and R-squared and let's say I am happy with results. Now I have 10 regression models (one for each fold) and if I want to score a new observation(s) I shall do:

  1. Produce 10 estimations for my new observation and then obtain an average estimation, or
  2. Average b0 and b1 to get a single model and then score my new observation

I think 1) is the right answer but I am not 100% sure.

$\endgroup$

1 Answer 1

3
$\begingroup$

The usual approach is to fit your final model to the full dataset, but report the cross-validated error (average of RMSE, MAE, R2, etc. across the folds). The hypothesis here is that when you use the full model to predict new data, the error should be similar to that you saw during cross-validation.

$\endgroup$
2
  • $\begingroup$ how you get from 10 models to your "final" model? $\endgroup$
    – Altons
    Nov 23, 2011 at 16:13
  • 2
    $\begingroup$ @Altons: You fit an 11th model, using the full dataset. Under 10-fold cross-validation, each model is fit to 9/10 of the data, and error is estimated using the remaining 1/10. You then fit a final model on 10/10 of the data (i.e. all of the data). $\endgroup$
    – Zach
    Nov 23, 2011 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.