Leave-one-out Cross Validation: What are best practices for reporting results and developing a final model? I am developing a simple linear regression model based on a limited set of data that represents monthly meteorological observations over six years (72 points). Of these data, I need a certain number of non-zero observations to make a reasonable regression. Furthermore, I need to capture seasonal variation so want to use complete years of data. So, the actual data available for the regression may be relatively small - I have set a minimum threshold of 11 nonzero observations to make the regression; if there are fewer, I don't make a model.
In order to make maximum use of the training data, I would use leave-one-out cross validation (LOOCV) to evaluate the quality of my regression model. For this, I leave out one year's worth of data at a time, fit the regression model with the remaining data, and test the resulting model against the excluded observations. This procedure results in six regression models and six RMSE estimates.
My questions are:

*

*In generating the final regression model, should I
average the fitted model coefficients across the six LOOCV
iterations? Or make a seventh iteration that uses all training data?
Or something else?

*What is best practice for reporting the quality of the final fitted model?
For example, report the RMSE for each LOOCV iteration separately, or
take a mean? Or something else?

 A: The common way to report the prediction error when using cross-validation (CV) is to gather the results from each fold, and then calculate whatever you had in mind, like the RMSE, on all of them. In your example, if I understand you correctly, the 12 first estimated outcomes, $\hat{y}_1,  \ldots, \hat{y}_{12}$ will be from the first fold, then the next 12, $\hat{y}_{13},  \ldots, \hat{y}_{24}$ will be from the second fold, and so on. Just plug them into your standard MSE
$$
\frac{1}{n}\sum_{i = 1}^n (y_i - \hat{y}_i)^2.
$$
In addition, you might want to report the mean and the standard deviation, because the standard deviation says something about the robustness of your model, which is usually of great interest.
If there is no particular reason not to, all training data should go into the final fitted model. The estimated coefficients have corresponding confidence intervals, which quantify the uncertainty in the estimation.
In general, there are a lot of pitfalls when using CV, and I recommend this book, freely available online, https://web.stanford.edu/~hastie/ElemStatLearn/printings/ESLII_print12_toc.pdf, for an introduction to CV.
