Which Kth model for prediction when training using fkolds?

Question

I think I understand the concept of cross validation. I built a model using kfolds and assessed it's performance based on the mean absolute error of n observations after having iterated test/train on ten folds.

Within each iteration I train a model then test it. So there are in fact k models.

Which one do I use for prediction?

# cross validation
for ( f in folds ) {
  train <- ptrain[-f,]
  test <- ptrain[f,]

  model <- lm(paste("loss ~ ",paste(predictors, collapse="+"),sep=""), data=train)
  predictions <- predict(model, interval="prediction", newdata=test)

  temp <- as.data.frame(predictions)

  cv_prediction <- rbind(cv_prediction, temp)
  testsetCopy <- rbind(testsetCopy, test)
}

After running this kfold loop, the model is just the last iteration of train. If I wanted to apply my model, what is my model? Trained on all available data?

Answer 1

With K-fold cross validation, you split the data in K parts, each time training a model on all but one of the parts, predicting the remaining part.

You state that you are fitting K different models in this process, but that is missing the point of cross-validation. The idea is that with cross-validation, you perform the selection of the model. In other words, you find out which of the covariates have sufficient contribution to the estimator. So while yes, you are fitting a model K times, the idea is to find out how good the specification of that model is, not to find the parameters of that model.

So what you should do, is in each of your iterations, instead of fitting just one linear model, fit all the possible combinations (or some reasonable subset) of covariates, instead of just the full model as you do now. This way you should get a range of predictions based on a bunch of different models.

You can then compare each of the models by checking the Mean Squared Error for the predictions and for every model: $MSE = \frac{1}{N}\sum_i^N (y_i - \hat{y_i})^2$. After the crossvalidation, you pick the model with the lowest MSE and fit that on the entire dataset to find its parameters.