As this question and its answer pointed out, k-fold cross validation (CV) is used for model selection, e.g. choosing between linear regression and neural network. It's also suggested that after deciding on which kind of model to use, the final predictor should be trained with the entire data set. Here my question is: how can we evaluate the final predictor? Is it sufficient to just use the average of the k accuracies obtained during k-fold CV?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
When training on each fold (90%) of the data, you will then predict on the remaining 10%. With this 10% you will compute an error metric (RMSE, for example). This leaves you with: 10 values for RMSE, and 10 sets of corresponding predictions. There are 2 things to do this these results:
Inspect the mean and standard deviation of your 10 RMSE values. k-fold takes random partitions of your data, and the error on each fold should not vary too greatly. If it does, your model (and its features, hyper-parameters etc.) cannot be expected to yield stable predictions on a test set.
Aggregate your 10 sets of predictions into 1 set of predictions. For example, if your training set contains 1,000 data points, you will have 10 sets of 100 predictions (10*100 = 1000). When you stack these into 1 vector, you are now left with 1000 predictions: 1 for every observation in your original training set. These are called out-of-folds predictions. With these, you can compute the RMSE for your whole training set in one go, as rmse = compute_rmse(oof_predictions, y_train). This is the likely the cleanest way to evaluate the final predictor.
-
$\begingroup$ What if I find a model with a high mean accurarcy during CV but also with a high accuracy variance over the folds? Then the model might not be good to use in production. Let's say I have another model with lower mean accuracy and also lower variance, then I should use this one. But how to automatically combine mean and variance into one scalar metric that just can be sorted to find the best model? Dividing the mean by the variance comes into my mind, but I am not sure if that is safe to do because of different scalings. $\endgroup$ Sep 6, 2020 at 13:33
$\begingroup$
$\endgroup$
Taking the average of the k accuracy scores is a macro-average. Taking the average of the 1000 individual predictions (as described in cavaunpeu's answer) is a micro-average. Both are legitimate and should be broadly similar to each other, so you can use whichever is more convenient - or report both for completeness.
