How to choose a predictive model after k-fold cross-validation?

Question

I am wondering how to choose a predictive model after doing K-fold cross-validation.

This may be awkwardly phrased, so let me explain in more detail: whenever I run K-fold cross-validation, I use K subsets of the training data, and end up with K different models.

I would like to know how to pick one of the K models, so that I can present it to someone and say "this is the best model that we can produce."

Is it OK to pick any one of the K models? Or is there some kind of best practice that is involved, such as picking the model that achieves the median test error?

Answer 1

I think that you are missing something still in your understanding of the purpose of cross-validation.

Let's get some terminology straight, generally when we say 'a model' we refer to a particular method for describing how some input data relates to what we are trying to predict. We don't generally refer to particular instances of that method as different models. So you might say 'I have a linear regression model' but you wouldn't call two different sets of the trained coefficients different models. At least not in the context of model selection.

So, when you do K-fold cross validation, you are testing how well your model is able to get trained by some data and then predict data it hasn't seen. We use cross validation for this because if you train using all the data you have, you have none left for testing. You could do this once, say by using 80% of the data to train and 20% to test, but what if the 20% you happened to pick to test happens to contain a bunch of points that are particularly easy (or particularly hard) to predict? We will not have come up with the best estimate possible of the models ability to learn and predict.

We want to use all of the data. So to continue the above example of an 80/20 split, we would do 5-fold cross validation by training the model 5 times on 80% of the data and testing on 20%. We ensure that each data point ends up in the 20% test set exactly once. We've therefore used every data point we have to contribute to an understanding of how well our model performs the task of learning from some data and predicting some new data.

But the purpose of cross-validation is not to come up with our final model. We don't use these 5 instances of our trained model to do any real prediction. For that we want to use all the data we have to come up with the best model possible. The purpose of cross-validation is model checking, not model building.

Now, say we have two models, say a linear regression model and a neural network. How can we say which model is better? We can do K-fold cross-validation and see which one proves better at predicting the test set points. But once we have used cross-validation to select the better performing model, we train that model (whether it be the linear regression or the neural network) on all the data. We don't use the actual model instances we trained during cross-validation for our final predictive model.

Note that there is a technique called bootstrap aggregation (usually shortened to 'bagging') that does in a way use model instances produced in a way similar to cross-validation to build up an ensemble model, but that is an advanced technique beyond the scope of your question here.

Answer 2

I found this excellent article How to Train a Final Machine Learning Model very helpful in clearing up all the confusions I have regarding the use of CV in machine learning.

Basically we use CV (e.g. 80/20 split, k-fold, etc) to estimate how well your whole procedure (including the data engineering, choice of model (i.e. algorithm) and hyper-parameters, etc.) will perform on future unseen data. And once you've chosen the winning "procedure", the fitted models from CV have served their purpose and can now be discarded. You then use the same winning "procedure" and train your final model using the whole data set.

Answer 3

Let me throw in a few points in addition to Bogdanovist's answer

As you say, you train $k$ different models. They differ in that 1/(k-1)th of the training data is exchanged against other cases. These models are sometimes called surrogate models because the (average) performance measured for these models is taken as a surrogate of the performance of the model trained on all cases.

Now, there are some assumptions in this process.

• Assumption 1: the surrogate models are equivalent to the "whole data" model.
  It is quite common that this assumption breaks down, and the symptom is the well-known pessimistic bias of $k$-fold cross validation (or other resampling based validation schemes). The performance of the surrogate models is on average worse than the performance of the "whole data" model if the learning curve has still a positive slope (i.e. less training samples lead to worse models).

• Assumption 2 is a weaker version of assumption 1: even if the surrogate models are on average worse than the whole data model, we assume them to be equivalent to each other. This allows summarizing the test results for $k$ surrogate models as one average performance.
  Model instability leads to the breakdown of this assumption: the true performance of models trained on $N \frac{k - 1}{k}$ training cases varies a lot. You can measure this by doing iterations/repetitions of the $k$-fold cross validation (new random assignments to the $k$ subsets) and looking at the variance (random differences) between the predictions of different surrogate models for the same case.

• The finite number of cases means the performance measurement will be subject to a random error (variance) due to the finite number of test cases. This source of variance is different from (and thus adds to) the model instablilty variance.

The differences in the observed performance are due to these two sources of variance.

The "selection" you think about is a data set selection: selecting one of the surrogate models means selecting a subset of training samples and claiming that this subset of training samples leads to a superior model. While this may be truely the case, usually the "superiority" is spurious. In any case, as picking "the best" of the surrogate models is a data-driven optimization, you'd need to validate (measure performance) this picked model with new unknown data. The test set within this cross validation is not independent as it was used to select the surrogate model.

You may want to look at our paper, it is about classification where things are usually worse than for regression. However, it shows how these sources of variance and bias add up.
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323

Answer 4

Why do we use k-fold cross validation?

Cross-validation is a method to estimate the skill of a method on unseen data. Like using a train-test split.

Cross-validation systematically creates and evaluates multiple models on multiple subsets of the dataset. This, in turn, provides a population of performance measures.

• We can calculate the mean of these measures to get an idea of how well the procedure performs on average.
• We can calculate the standard deviation of these measures to get an idea of how much the skill of the procedure is expected to vary in practice.

This is also helpful for providing a more nuanced comparison of one procedure to another when you are trying to choose which algorithm and data preparation procedures to use.

Also, this information is invaluable as you can use the mean and spread to give a confidence interval on the expected performance on a machine learning procedure in practice.

reference

Answer 5

It's a very interesting question. To make it clear, we should understand the difference of model and model evaluation. We use full training set to build a model, and we expect this model would be finally used.

K fold cross evaluation would build K models but all would be dropped. The K models are just used for evaluation. and It just produced metrics to tell you how well this model fits with your data.

For example, you choose LinearRegression algo and perform two operation on the same training set: one with 10 fold cross validation, and the other with 20 fold. the regression(or classifier) model should be the same, but Correlation coefficient and Root relative squared error are different.

Below are two runs for 10 fold and 20 fold cross validation with weka

1st run with 10 fold

=== Run information ===
Test mode:    10-fold cross-validation
...
=== Classifier model (full training set) ===


Linear Regression Model  <---- This model is the same

Date = 844769960.1903 * passenger_numbers -711510446549.7296

Time taken to build model: 0 seconds

=== Cross-validation ===  <---- Hereafter produced different metrics
=== Summary ===

Correlation coefficient                  0.9206
Mean absolute error                35151281151.9807
Root mean squared error            42707499176.2097
Relative absolute error                 37.0147 %
Root relative squared error             38.9596 %
Total Number of Instances              144     

2nd run with 20 fold

=== Run information ===
...
Test mode:    20-fold cross-validation

=== Classifier model (full training set) ===


Linear Regression Model   <---- This model is the same

Date = 844769960.1903 * passenger_numbers -711510446549.7296

Time taken to build model: 0 seconds

=== Cross-validation ===  <---- Hereafter produced different metrics
=== Summary ===

Correlation coefficient                  0.9203
Mean absolute error                35093728104.8746
Root mean squared error            42790545071.8199
Relative absolute error                 36.9394 %
Root relative squared error             39.0096 %
Total Number of Instances              144     

Answer 6

I am not sure the discussion above is entirely correct. In cross-validation, we can split the data into Training and Testing for each run. Using the training data alone, one needs to fit the model and choose the tuning parameters in each class of models being considered. For example, in Neural Nets the tuning parameters are the number of neurons and the choices for activation function. In order to do this, one cross-validates in the training data alone.

Once the best model in each class is found, the best fit model is evaluated using the test data. The "outer" cross-validation loop can be used to give a better estimate of test data performance as well as an estimate on the variability. A discussion can then compare test performance for different classes say Neural Nets vs. SVM. One model class is chosen, with the model size fixed, and now the entire data is used to learn the best model.

Now, if as part of your machine learning algorithm you want to constantly select the best model class (say every week), then even this choice needs to be evaluated in the training data! Test data measurement cannot be used to judge the model class choice if it is a dynamic option.

Answer 7

Even belatedly, let me throw in my 2 drachmas. I am of the opinion that you can train a model using KFold cross-validation as long as you do it inside a loop. I am not posting everything, just the juice. Mind you that the functions below can be used with any model, be it an sklearn predictor/pipeline, a Keras or a Pytorch model after affecting the necessary array/tensor conversions is some cases. Here goes:

#Get the necessary metrics (in this case for a severely imbalanced dataset)

def get_scores(model,X,y):
  pred=np.round(model.predict(X))
  probs=model.predict_proba(X)[:,1]
  precision,recall,_=precision_recall_curve(y,probs)
  accu=accuracy_score(y,pred)
  pr_auc=auc(recall,precision)
  f2=fbeta_score(y,pred,beta=2)
  return pred,accu,pr_auc,f2


#Train model with KFold cross-validation

def train_model(model,X,y):
  accu_list,pr_auc_list,f2_list=[],[],[]
  kf=StratifiedKFold(5,False,seed)
  for train,val in kf.split(X,y):
    X_train,y_train=X[train],y[train]
    X_val,y_val=X[val],y[val]
    model.fit(X_train,y_train)
    _,accu,pr_auc,f2=get_scores(model,X_val,y_val)
    accu_list.append(accu)
    pr_auc_list.append(pr_auc)
    f2_list.append(f2)
  print(f'Training Accuracy: {np.mean(accu_list):.3f}')
  print(f'Training PR_AUC: {np.mean(pr_auc_list):.3f}')
  print(f'Training F2: {np.mean(f2_list):.3f}')
  return model

So, after training and when you want to predict unknown data you would just do:

fitted_model=train_model(model,X,y)

I hope that works out for you. In my case it works everytime. Now, computational cost is different - and very sad - matter.

Answer 8

Consider two types of algorithms:

1. Algorithms come with hyperparameters, which do not change with different data subsets. Cross-validation might be used to evaluate

• the performance of different algorithms
• feature engineering
• feature input
• the selection of best hyperparameters.

However, with those algorithms (such as tree based methods), different data subsets in CV may have different splitting rules. Except the choice of your ML procedures and hyperparameter tuning, you need a final rule to be applied for your future data. Therefore, applying your final choices to the full dataset is normally required to obtain the final model.

2. Algorithms come with parameters that change with different data.

Statistical models such as (generalised) linear models might be easier to understand. CV might be used to evaluate

• the performance of different models
• the performance of different distributional assumptions
• feature engineering
• feature input

With such methods, after selecting a model and features going into the model based on CV, you need regression coefficients to be applied to the future data to make prediction.

Where are those coefficients coming from then?

One way is to apply the final model to your entire dataset to obtain them. The other way is to take average on coefficients obtained from different data subsets if the full data is too large to run in one go.

If you are using training/validation/test splits, then I would normally treat training + validation as the full dataset, and the test split as future data to test on your final model.