(I'm sure we have some answers of this question already, but I cannot find them right now.)
Does that mean I'm stuck with ensembling k models for all future predictions now?
No, usually that's not what you do.
Ensemble models differ fundamentally from models that do not aggregate the predictions of many submodels.
For "normal" (unaggregated) cross validation, you typically apply the same training algorithm that was used during cross validation to fit the surrogate models to the whole data set (as it is before splitting for cross validation).
In sklearn context, that means the
fit function of the
estimator you hand over to
With the example in
>>> from sklearn import datasets, linear_model
>>> from sklearn.model_selection import cross_val_predict
>>> diabetes = datasets.load_diabetes()
>>> X = diabetes.data[:150]
>>> y = diabetes.target[:150]
>>> lasso = linear_model.Lasso()
>>> y_pred = cross_val_predict(lasso, X, y, cv=3)
>>> final_model = lasso.fit (X, y)
>>> new_predictions = final_model.predict (diabetes.data [440:442])
Why not ensemble
Ensemble models differ from "single" not aggregated models by aggregating the predictions of several submodels. The aggregated model will yield more stable predictions than each of the submodels if the submodels suffer from instability. If the submodels are stable, there's no difference in the prediction (just a waste of computational resources).
Normal cross validation compares un-aggregated predictions to the ground truth, so it doesn't evaluate possible stabilization by aggregating.
- for an un-aggregated model, an un-aggregated (i.e. the usual) cross validation can be used as approximation for predictive performance/generalization error estimate.
- for an ensemble model, we also need an ensemble-type estimation of performance / generalization error, such as out-of-bag or its cross validation analogue.
And vice versa.
Using an un-aggregated cross validation estimate for an ensemble model will cause a pessimistic bias that can be anywhere between negligible and large, depending on how stable the CV surrogate models are and how many surrogate models are aggregated.
Moreover, this bias is unnecessary as using the corresponding fit function and cross validaton scheme will only be subject to the small pessimistic bias for having feweer training cases (and/or fewer submodels to aggregate) for the surrogate models.