10
$\begingroup$

I have a model trained as follows.

random_search=RandomizedSearchCV(classifier, param_distributions=params,\
     scoring='f1',
     n_iter=100, cv=5, verbose=3, random_state=42, n_jobs=-1)
random_search.fit(X, y)
final_classifier=random_search.best_estimator_

My question is, do I have to train final_classifier on entire data before putting it into production?

i.e. Should I do

final_clssifier.fit(X,y)

Or random_search returning the model after fitting it again on entire data? I really appreciate any help you can provide.

Improve this question
$\endgroup$
2
  • $\begingroup$ I recommend fharrell.com/post/split-val $\endgroup$
    – Oliver882
    Aug 23, 2022 at 7:03
  • $\begingroup$ @Oliver882 What is their recommended method then, instead of split-sample validation? They mention bootstrapping. Where can I read more details about it? $\endgroup$
    – Jessica
    Aug 28, 2022 at 14:18

4 Answers 4

Reset to default
7
$\begingroup$

The argument refit is for this purpose, which is by default True.

See the documentation:

refit: bool, str, or callable, default=True

Refit an estimator using the best found parameters on the whole dataset.

About putting it into the production, I'd suggest you separate some part of your dataset as a test set and do a final check with the refitted model. Although CV is there to select best HPs, it doesn't have the ability to prevent overfitting, and it's for your good to detect if you have it and choose to go with it. Nobody should go to production blindfold.

Improve this answer
$\endgroup$
2
  • 4
    $\begingroup$ +1, but "it doesn't have the ability to prevent overfitting" doesn't seem quite right. The HPs selected have best out-of-fold performance among those searched, so will tend to be less overfit or at least perform well on unseen (iid!) data well despite overfitting. It is true that the best_score_ should not be taken as representative of performance though, due to selection bias, so assuming such a score is desired absolutely keep an additional test set (or nested CV). $\endgroup$
    – Ben Reiniger
    Aug 22, 2022 at 13:14
  • 3
    $\begingroup$ Thanks for your comment @BenReiniger, that is right. It has mitigating effects, but I just wanted to underline that doing cross validation to select HPs does not make the model (to be trained only with training set) overfit-proof, which is a common misconception among ML practitioners. $\endgroup$
    – gunes
    Aug 22, 2022 at 17:13
7
$\begingroup$

It is not required. In many cases, it might be a good idea, for example when you have a rather small dataset. On the other hand, if you have a lot of good data, so the set used for tuning was already decent, it might not be necessary. It not always will be possible as well, for example, if training your model cost as much as a new car (some large deep learning models) you might not have enough resources to repeat it, same if the training takes very long and you don't have the time. If you decide to re-train, it is a good idea to keep the held-out test set to make sure that the re-trained model performs as well as the one found during cross-validation, as there's always a risk that something goes wrong (e.g. bug in the code, for a trivial example).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Tim RandomizedSearchCV returns the model after refitting the data in the whole data as suggested by @gunes. So it is not required in this case, right? $\endgroup$
    – NAS_2339
    Aug 22, 2022 at 8:04
  • 1
    $\begingroup$ @NAS_2339 I am answering if it is "required" in the sense "if you should do it" not if it was done for you, but gunes is correct. Still, as said in the answer, you probably need a held-out test set to verify the performance of the final model. $\endgroup$
    – Tim ♦
    Aug 22, 2022 at 8:07
  • 1
    $\begingroup$ Comment: (+1) but unless it is rather large, that "hold-out set" is more like a canary in the coal mine rather than a good indication of generalisation performance. Having resampling-based performance estimates is more realistic about generalisation performance and its variability. $\endgroup$
    – usεr11852
    Aug 22, 2022 at 16:15
  • $\begingroup$ Definitely. It's just when you need to deploy a single model, it's the resampling of the test set we're only left with. $\endgroup$
    – gunes
    Aug 22, 2022 at 20:09
3
$\begingroup$

If you do cross-validation (CV), you have one model per fold. E.g. if you did 2-times-repeated 7-fold CV, you have 14 models. Sometimes it is acceptable to have that many models and to just average their results.

However, you reduce runtime (just one model), make it easier to use model interpretability tools and may also get better performance, if you refit on all the data. Whether you will is not always totally clear, you on the one hand have more data (ought to improve things), but by training multiple times on different data, you are getting some (potentially useful) variety and are to some extent doing seed averaging (you can of course train "a final time" on all data multiple times with different random number seeds to get that same effect). From what I've heard from top-Kagglers it's more likely that retraining on all data is the best choice.

What are reasons not to? Training cost is probably not the main issue, since you could afford to do CV, so just one more fit presumably does not matter so much. However, sometimes you do early stopping based on CV. Generally, you'd want to define when you stop based on CV (e.g. identify a fixed number of epochs/trees/whatever that seems to work across folds), but if your training approach is so fragile that this is not possible and you need a validation set to decide when to stop (arguably, you ought to stabilize things to prevent this situation), that could be an argument for avoiding re-training.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ "Sometimes it is acceptable to have that many models and to just average their results" that would in fact be an ensemble model and different from the model trained on the whole data set. Also, its predictive performance is not what is estimated by the usual CV performance estimates. $\endgroup$
    – cbeleites unhappy with SX
    Aug 22, 2022 at 13:37
  • $\begingroup$ @cbeleites And what the usual CV performance estimates might not be what many think it estimates: Cross-validation: what does it estimate and how well does it do it?. So it may be reasonable to conclude that no one strategy is guaranteed to be optimal. $\endgroup$
    – dipetkov
    Aug 22, 2022 at 20:40
  • $\begingroup$ @dipetkov: I'm very much aware of that paper :-) - and actually currently trying to work out how that aligns with the perspective I take on CV (where I look at surrogate models and samples as two distinct factors/sources of variance). And I may have to update some old answers of mine which may be inadvertently misleading. BUT... ensemble prediction is yet a very different beast from the question whether CV estimates are more closely related to performance of the model trained on all available data or the set of models trained from data sets drawn from the training data population. $\endgroup$
    – cbeleites unhappy with SX
    Aug 24, 2022 at 9:49
  • $\begingroup$ The average of certain fold-wise error estimates is not the same as the error estimate of predictions averaged over those folds. (In general, it can be in certain situations such as 0 error in the folds) $\endgroup$
    – cbeleites unhappy with SX
    Aug 24, 2022 at 9:51
  • 1
    $\begingroup$ I'm a bit puzzled by the comments. If you do not re-train on all training data and stay with what you created during cross-validation, surely you have to either (a) pick the model from a single fold (but how would you choose, given that you have nothing to base such a choice on since the out-of-fold performance cannot really be compared across folds, and doing so in a way would negate the whole point of CV), or (b) average (probably equally weighted, because, again, how would you choose weights?) the outputs from the models for each fold. $\endgroup$
    – Björn
    Aug 24, 2022 at 11:30
1
$\begingroup$

It slightly depends on what you mean by "entire data".

In the "train, cross-validate, test" paradigm, where the cross-validation folds are taken from the training set, once you have selected your model and hyper-parameters you should then train that model on the full training set and finally (and only once) test it against the test set you set aside at the beginning to see how good this final model is on unseen data.

Whether you then

(a) use that tested model in production, or

(b) retrain it (with the same model and hyper-parameters) on the combined training and test set

is your choice. If you have sufficient data and your test data was similar to your training set, then it should not make much difference. You would not have a test of the accuracy of an option (b) model on out-of-sample data until you have new previously unseen data, but that may not be an issue and you may want the small theoretical benefit of training on slightly more data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.