In leave one out cross validation, if I have 10 data points, and I am trying to fit a linear regression model (as an example), then I generate 10 different models, each time using 9 data points and testing the model on the 10th data point.
After that, do I just select the model that gave the lowest error? or do I take an average of the coefficients of the 10 different models for my final model?
Cross validation gives you an idea of how reliable your model will be in data it wasn't trained on. In practice, i look at the performance characteristic using some appropriate metric on the residuals of the holdout sets. Then once you're satisfied that the model type selected is appropriate, you train the model on all the data to put into production. Selecting just one of your hold-out trials wouldn't give you any information on future performance.
Here's an illustration. Imagine a linear regression using n-fold CV like you're describing. Put one major outlier in the data. The poorest performing trial would be the one with the outlier in the holdout test set. But since outliers are rare, it's actually a better model for production because it's trained on data that it's likely to see in practice.
So we typically don't use CV to pick a single trial, but to test different models, input variables, etc.
