6
$\begingroup$

I want to do the following steps:

  1. Train a model with cross-validation
  2. Use that model for future predictions (including my test set)

cross_val_predict only gives me its predictions for the training set. After that, where is my classifier to keep for future predictions? The closest I've found is to run cross_validate and set return_estimator=True. This gives me an array of its trained estimators, since it creates a separate instance for each fold.

Does that mean I'm stuck with ensembling k models for all future predictions now? If so, is there a convenient way to do that? Using the VotingClassifier perhaps? What's the best practice here?

$\endgroup$
6
$\begingroup$

tl;dr Yes, you could build a voting classifier using the array of your trained estimators. I don't recommend that, especially as I don't know much about the use case.

My typical workflow

  • There could be multiple candidates to build a decent estimator in this use case (Let's say RandomForest, xgboost, SVM)
  • Build an estimator using each of them with their optimal hyper parameters (here is one way to do that)
  • Evaluate the above estimators for predictive power and how well they generalize across different test sets (I would use Cross-validation here)
  • Some/all candidate estimators seem to fair well (this would be helpful here)
  • Build an ensemble of multiple estimators for better predictive and/or generalizing power. (I could use VotingClassifier here)

VotingClassifier is mainly used to vote among different techniques, you could of course also use it as you said though. Following is a quick take on usage of cross validation and also about voting.


Thoughts about your use case

Cross-validation is mainly used as a way to check for over-fit. Assuming you have determined the optimal hyper parameters of your classification technique (Let's assume random forest for now), you would then want to see if the model generalizes well across different test sets.

Cross-validation in your case would build k estimators (assuming k-fold CV) and then you could check the predictive power and variance of the technique on your data as following:

  1. mean of the quality measure. Higher, the better

  2. standard_deviation of the quality measure. Lower, the better A high mean and low standard deviation of your quality measure would mean the modeling technique is doing well.

Assuming the above measure looks good, you could then conclude that random forest with the hyper parameters used is a decent candidate model. If you think your classification technique does well enough, then you could also build a voting classifier using the k estimators from CV.

$\endgroup$
4
  • 2
    $\begingroup$ Thank you! They key thing I was missing is that CV is not a method for training a model. It's only for evaluation. With the help of CV, you can assess hyperparameters and compare different models to each other. It's just an alternative to a train/test split evaluation. $\endgroup$ – Steven Jun 3 '19 at 8:28
  • $\begingroup$ @Steven Please mark the answer as accepted if this was helpful. That way, it has more use for others who might face this issue in future. $\endgroup$ – Sunil Jun 3 '19 at 9:00
  • $\begingroup$ I'm sorry to contradict: an ensemble model is fundamentally different from a "single" model without aggregatiom. For the ensemble model, also "normal" cross validation which doesn't do any aggregation to arrive at the predictions does not yield a good estimate of the ensemble model's performance. For that you'd use the CV-analogue of the out-of-bag estimate (see e.g. our paper Beleites & Salzer: Assessing and improving the stability of chemometric models in small sample size situations AnalBioanalChem 2008, 1261-1271. DOI: dx.doi.org/10.1007/s00216-007-1818-6 . $\endgroup$ – cbeleites unhappy with SX Jun 3 '19 at 15:16
  • $\begingroup$ The crucial point here is that normal cross validation ignores the stabilization effect of the aggregation (which is the whole point of aggregating). So normal cross validation can be used as approximation/estimate for generalization error of an unaggregated model only. That is usually the same training function as used during cross validation applied to the whole data set. $\endgroup$ – cbeleites unhappy with SX Jun 3 '19 at 15:18
1
$\begingroup$

(I'm sure we have some answers of this question already, but I cannot find them right now.)

Does that mean I'm stuck with ensembling k models for all future predictions now?

No, usually that's not what you do.

Ensemble models differ fundamentally from models that do not aggregate the predictions of many submodels.

For "normal" (unaggregated) cross validation, you typically apply the same training algorithm that was used during cross validation to fit the surrogate models to the whole data set (as it is before splitting for cross validation).

In sklearn context, that means the fit function of the estimator you hand over to cross_validate:

With the example in cross_val_predict:

>>> from sklearn import datasets, linear_model
>>> from sklearn.model_selection import cross_val_predict
>>> diabetes = datasets.load_diabetes()
>>> X = diabetes.data[:150]
>>> y = diabetes.target[:150]
>>> lasso = linear_model.Lasso()
>>> y_pred = cross_val_predict(lasso, X, y, cv=3)

Now,

>>> final_model = lasso.fit (X, y)
>>> new_predictions = final_model.predict (diabetes.data [440:442])

Why not ensemble

Ensemble models differ from "single" not aggregated models by aggregating the predictions of several submodels. The aggregated model will yield more stable predictions than each of the submodels if the submodels suffer from instability. If the submodels are stable, there's no difference in the prediction (just a waste of computational resources).

Normal cross validation compares un-aggregated predictions to the ground truth, so it doesn't evaluate possible stabilization by aggregating.

Thus,

  • for an un-aggregated model, an un-aggregated (i.e. the usual) cross validation can be used as approximation for predictive performance/generalization error estimate.
  • for an ensemble model, we also need an ensemble-type estimation of performance / generalization error, such as out-of-bag or its cross validation analogue.

And vice versa.

Using an un-aggregated cross validation estimate for an ensemble model will cause a pessimistic bias that can be anywhere between negligible and large, depending on how stable the CV surrogate models are and how many surrogate models are aggregated.
Moreover, this bias is unnecessary as using the corresponding fit function and cross validaton scheme will only be subject to the small pessimistic bias for having feweer training cases (and/or fewer submodels to aggregate) for the surrogate models.

$\endgroup$
2
  • $\begingroup$ Seems like the consensus is to validate using CV, and then call fit so that it can be used for predictions on new data. My only issue with that is that presumably fit is being called at least 1 more time than necessary, and I'm a cheap jerk. $\endgroup$ – Chris Mar 10 at 14:13
  • 1
    $\begingroup$ @Chris: yes, but it's called on a larger training set, so you get the chance to a (slightly) better model as return for the computation invested. $\endgroup$ – cbeleites unhappy with SX Mar 10 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.