How to use a cross-validated model for prediction?

Question

I want to do the following steps:

1. Train a model with cross-validation
2. Use that model for future predictions (including my test set)

cross_val_predict only gives me its predictions for the training set. After that, where is my classifier to keep for future predictions? The closest I've found is to run cross_validate and set return_estimator=True. This gives me an array of its trained estimators, since it creates a separate instance for each fold.

Does that mean I'm stuck with ensembling k models for all future predictions now? If so, is there a convenient way to do that? Using the VotingClassifier perhaps? What's the best practice here?

Answer 1

tl;dr Yes, you could build a voting classifier using the array of your trained estimators. I don't recommend that, especially as I don't know much about the use case.

My typical workflow

• There could be multiple candidates to build a decent estimator in this use case (Let's say RandomForest, xgboost, SVM)
• Build an estimator using each of them with their optimal hyper parameters (here is one way to do that)
• Evaluate the above estimators for predictive power and how well they generalize across different test sets (I would use Cross-validation here)
• Some/all candidate estimators seem to fair well (this would be helpful here)
• Build an ensemble of multiple estimators for better predictive and/or generalizing power. (I could use VotingClassifier here)

VotingClassifier is mainly used to vote among different techniques, you could of course also use it as you said though. Following is a quick take on usage of cross validation and also about voting.

---

Thoughts about your use case

Cross-validation is mainly used as a way to check for over-fit. Assuming you have determined the optimal hyper parameters of your classification technique (Let's assume random forest for now), you would then want to see if the model generalizes well across different test sets.

Cross-validation in your case would build k estimators (assuming k-fold CV) and then you could check the predictive power and variance of the technique on your data as following:

1. mean of the quality measure. Higher, the better

2. standard_deviation of the quality measure. Lower, the better A high mean and low standard deviation of your quality measure would mean the modeling technique is doing well.

Assuming the above measure looks good, you could then conclude that random forest with the hyper parameters used is a decent candidate model. If you think your classification technique does well enough, then you could also build a voting classifier using the k estimators from CV.

Answer 2

(I'm sure we have some answers of this question already, but I cannot find them right now.)

Does that mean I'm stuck with ensembling k models for all future predictions now?

No, usually that's not what you do.

Ensemble models differ fundamentally from models that do not aggregate the predictions of many submodels.

For "normal" (unaggregated) cross validation, you typically apply the same training algorithm that was used during cross validation to fit the surrogate models to the whole data set (as it is before splitting for cross validation).

In sklearn context, that means the fit function of the estimator you hand over to cross_validate:

With the example in cross_val_predict:

>>> from sklearn import datasets, linear_model
>>> from sklearn.model_selection import cross_val_predict
>>> diabetes = datasets.load_diabetes()
>>> X = diabetes.data[:150]
>>> y = diabetes.target[:150]
>>> lasso = linear_model.Lasso()
>>> y_pred = cross_val_predict(lasso, X, y, cv=3)

Now,

>>> final_model = lasso.fit (X, y)
>>> new_predictions = final_model.predict (diabetes.data [440:442])

---

Why not ensemble

Ensemble models differ from "single" not aggregated models by aggregating the predictions of several submodels. The aggregated model will yield more stable predictions than each of the submodels iff the submodels suffer from instability. If the submodels are stable, there's no difference in the prediction (just a waste of computational resources).

Normal cross validation compares un-aggregated predictions to the ground truth, so it doesn't evaluate possible stabilization by aggregating.

Thus,

• for an un-aggregated model, an un-aggregated (i.e. the usual) cross validation can be used as approximation for predictive performance/generalization error estimate.
• for an ensemble model, we also need an ensemble-type estimation of performance / generalization error, such as out-of-bag or its cross validation analogue.
  For details, see e.g. our paper Beleites & Salzer: Assessing and improving the stability of chemometric models in small sample size situations AnalBioanalChem 2008, 1261-1271. DOI: 10.1007/s00216-007-1818-6

And vice versa.

Using an un-aggregated cross validation estimate for an ensemble model will cause a pessimistic bias that can be anywhere between negligible and large, depending on how stable the CV surrogate models are and how many surrogate models are aggregated.
Moreover, this bias is unnecessary as using the corresponding fit function and cross validaton scheme will only be subject to the small pessimistic bias for having feweer training cases (and/or fewer submodels to aggregate) for the surrogate models.