Making ensemble of K models trained during K-fold cross-validation

Question

Let's suppose we are doing K-fold cross-valiation to estimate the performance of a model with a given set of hyperparameters.

X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([0, 0, 1, 1])
model = RandomForestClassifier()
rkf = ShuffleSplit(n_splits=5, random_state=42)
models_list = []
for train_index, test_index in rkf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]


    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    accs = accuracy_score(y_test, preds)
    models_list.append(model)

ensemble_model = VotingClassifier(models_list) # Ensemble the models into a VotingClassifier

If I'm not mistaken, the models trained on each fold are not combined into an ensemble to be used in production. Can we combine the models trained on the individual folds in an ensemble and use that in production?

Answer 1

The answer is that yes, this is possible. This is addressed in the StackOverflow answer here: https://stackoverflow.com/a/28508619/6479831. The "model addition" is performed by modifying the estimators_ and n_estimators attributes of the models.

The following code block is copied from the answer linked above, and demonstrates how to do this.

from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import train_test_split
from sklearn.datasets import load_iris

def generate_rf(X_train, y_train, X_test, y_test):
    rf = RandomForestClassifier(n_estimators=5, min_samples_leaf=3)
    rf.fit(X_train, y_train)
    print "rf score ", rf.score(X_test, y_test)
    return rf

def combine_rfs(rf_a, rf_b):
    rf_a.estimators_ += rf_b.estimators_
    rf_a.n_estimators = len(rf_a.estimators_)
    return rf_a

iris = load_iris()
X, y = iris.data[:, [0,1,2]], iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.33)
# in the line below, we create 10 random forest classifier models
rfs = [generate_rf(X_train, y_train, X_test, y_test) for i in xrange(10)]
# in this step below, we combine the list of random forest models into one giant model
rf_combined = reduce(combine_rfs, rfs)
# the combined model scores better than *most* of the component models
print "rf combined score", rf_combined.score(X_test, y_test)

However, see my comment on your original question for notes regarding whether this is advisable to do.

Edited to add details about what this does: A random forest is just a large ensemble of trees. Each of your K-folds produces such an ensemble. The code shown above simply combines all of these trees into one large ensemble. This is in principle not different from ordinary training of a full random forests model, since random forests train each tree on a bagged version of the data; K-fold CV essentially enforces constraints on the bagging that is used to build your trees. I suspect that in general you would see fairly modest improvements from a model trained on the full dataset in comparison with a model built from combining ensembles built under K-folds (all else equal).

Answer 2

There's two versions of this:

1. You trained a model (=just a single model) on K-folds to estimate performance. You don't want to re-train on all data (e.g. because that makes early stopping based on out-of-fold performance difficult). You can just average the predictions of all the models (e.g. average of predicted continuous values, average of probabilities for binary or multi-class prediction, but also voting ensemble and so on). However, note that there's not really any good reason to deviate from a simple average of some form and give more weight to one version of a model over another trained on different folds (you cannot really compare the out-of-fold performances). There's no particular reason this would be wrong, although I have heard a former Kaggle #1 mention that re-training on all data usually improves performance. Nevertheless, this is sometimes done e.g. for Kaggle. In practice, retraining on all data is also interesting to reduce complexity.
2. You train multiple models (e.g. same model type, different hyperparameters, or different model classes) for each fold and use the out-of-fold predictions to decide how to combine their predictions (e.g. using some simple model to combine them, or just optimizing the weights in a weighted average) with hyperparameters tuned using the same CV scheme as before ("model stacking" or ensembling). Thereafter, you can - again - either combine for the models per fold or re-train on all data and then combine. All the same considerations as in point 1) apply, just with one extra layer of ensembling.

Answer 3

Can we combine the models trained on the individual folds in an ensemble and use that in production?

Yes. If you need a reference we've been doing this in Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6

However, for models that are already ensembles like randomForest, this isn't really substantially different from a randomForest with more trees.

I wasnt sure if theres is a mathematical reason for not doing it.

(For randomForest: see above)

There are conditions when it improves predictive ability and others where it doesn't:

If the surrogate models are stable in their predictions (you can easily and directly measure this by repeated k-fold CV), the ensemble will predict just as the single models and no better.
If the surrogate models are somewhat unstable, that instability (variance) is reduced by the ensemble prediction. The ensemble prediction cannot help with bias in the predictions.

Side remark: a consequence of this is that I wouldn't expect hyperparameter settings optimized for single model prediction to be optimal for ensemble prediction.
(However, in practice you may find that they work well, because we often use hyperparameter tuning heuristics that err on the side of high complexity with too low bias and too high variance for single model prediction - which should be better for ensemble prediction)

---

The above was written with a fixed/single hyperparameter set for all models in the ensemble in mind.

If you look at building ensembles across e.g. a range of different complexities, have a look at Bayesian model averaging as well.

Answer 4

nnUNet is a successful procedure for automatic hyperparameter tuning and optimization of UNets (neural segmentation models), which applies the suggested method, https://arxiv.org/abs/1809.10486: "For the test cases we use the five networks obtained from our training set cross-validation as an ensemble to further increase the robustness of our models." I don't believe they claim it's optimal, but maybe the approach has more merit for deep neural nets than for random forest models?