5
$\begingroup$

My question is a follow-up of posts such as this.

To summarise, I am given say 1000 data points. I want to fit say Random Forest and optimise among $n$ choices of hyperparameters using k-fold cross-validation. I have trained $n \times k$ models and can select the best choice among the $n$ hyperparameters sets (having $k$ surrogate models per set of hyperparameters). So far so good.

Now that I have selected near-optimal hyperparameters, I can re-train the model on my whole dataset. There is no validation set though, I have to wait for a real data point coming from the wild and be confident my model is the best possible.

The question is: how do I know my model is the "best possible"? Could it not be that I have overfitted when training on the whole dataset, and would not have been better to retain one (assuming there was a way to select) of the $k$ surrogate models using the best set of hyperparameters? Two questions then:

  • Is there a rigorous justification for the procedure sketched above?
  • Is one needed at all, and the argument "more data -> better prediction when facing a new datapoint" is obvious?

Thanks

$\endgroup$
1
  • 1
    $\begingroup$ You have used the "train/cross-validate" paradigm to choose what you hope is the best possible model with the best (or good enough) hyperparameters. So the only way you can test it is on new previously unseen data. Even that is not a "best possible" relative result, but simply an absolute assessment of performance on unseen data. You could have originally followed a "train/cross-validate/test" paradigm by holding back some of the original data to be a test set, doing what you did for your final model on the rest of the data, and then testing it once on the held-out data. $\endgroup$
    – Henry
    Oct 5, 2023 at 12:38

2 Answers 2

6
$\begingroup$

how do I know my model is the "best possible"?

We can't. Cross-validation and other model selection approaches are useful tools, but none of them are perfect. Generalizing from a finite sample is inherently difficult. CV can do a decent job of weeding out unsuitable models, but cannot guarantee that you chose the "best" or "true" model even if there is one.

Could it not be that I have overfitted when training on the whole dataset, and would not have been better to retain one (assuming there was a way to select) of the k surrogate models using the best set of hyperparameters?

It depends! There's under/overfit in the choice of hyperparameters during CV itself, and then there's under/overfit in the model parameters and predictions on new data.

If you plan to re-train on the whole dataset, it's actually possible that CV will under-fit the hyperparameters. Your surrogate models were trained on samples of size $n*(k-1)/k$, not $n$. So they were trying to find optimal hyperparameters for datasets smaller than your whole dataset. With smaller datasets, we can only afford to fit less-flexible models, so they likely chose hyperparameters that are less flexible (i.e. more under-fitting) than the optimal ones for your full dataset.

On the other hand, choosing the hyperparameters which optimize performance is a noisy problem. We are trying to find the location of the minimum (or maximum) of a curve, when there's noise in the height of the curve, which means that the apparent min (or max) is not the true one. And the error is not equally likely in either direction: People have shown that the hyperparameters that "win" on a particular CV run are more likely to be too flexible (i.e., over-fitting) for samples of size $n*(k-1)/k$. But that's about CV's choice of hyperparameters -- not about re-training on the whole dataset.

In short: CV often chooses hyperparameters that overfit slightly for samples of size $n*(k-1)/k$, but might still underfit for samples of size $n$.

Yet, regardless of what hyperparameters CV chooses: when you fit a model with those chosen parameters to a larger dataset, it ought to reduce variance in the predictions without increasing bias.

Is there a rigorous justification for the procedure sketched above? Is one needed at all, and the argument "more data -> better prediction when facing a new datapoint" is obvious?

The argument is "obvious" in the sense that more data leads to better predictions on average. More data will reduce the variance in fitting your chosen model & consequently reduce the variance in the fitted model's predictions, without increasing the bias (assuming your data really are representative of the population). That said, we cannot guarantee it will be true in individual instances.

$\endgroup$
7
  • $\begingroup$ Thanks, this is the answer I was hoping for. Is it then correct to say that on average re-training on the whole dataset is the best strategy? I am asking as you start a paragraph with "If you plan to re-train on the whole dataset <...>" but then from your arguments, seems there no other rational option, to maximise success chances, did I get it right? $\endgroup$
    – Smerdjakov
    Oct 5, 2023 at 13:58
  • $\begingroup$ There's no one "best" strategy. Your strategy is: "Keep it all as one dataset. Run K-fold CV, then use the chosen hyperparameters and re-train the model on the whole dataset." But another strategy as @Henry suggested could be: "Set aside a holdout set, and use the rest for training. Run K-fold CV on the training set, use the chosen hyperpars to re-train the model on the training set, and then evaluate that model's performance on the holdout set." This uses less data for final training, but gives you a more unbiased estimate of actual performance. So it depends on whether you need that or not. $\endgroup$
    – civilstat
    Oct 5, 2023 at 17:09
  • $\begingroup$ Another strategy is nested CV, whose estimates of model performance can be less variable than the holdout-set strategy. Yet another is: Don't do CV at all, and instead use measures like AIC or BIC for model selection. Each has its pros and cons. $\endgroup$
    – civilstat
    Oct 5, 2023 at 17:15
  • $\begingroup$ For a deeper dive into differences between all these strategies, take a look at Julian Faraway's blog post "Is data splitting good for you?" and arXiv paper "Does data splitting improve prediction?". $\endgroup$
    – civilstat
    Oct 5, 2023 at 17:19
  • $\begingroup$ I get it there is no "best strategy" in general, I was referring to my approach only.There are many strategies, but assuming CV is run to select hypers on all the data, is it reasonable to say that, for the next prediction on new datapoints, it is on average best to retrain on the whole dataset as, according to your last paragraph, it will reduce variance without affecting bias (excluding underfit/overfit in the choice of hypers, just talking about the effect of eventually considering more data). Sorry if I am being dense I find the topic slippery, thanks $\endgroup$
    – Smerdjakov
    Oct 5, 2023 at 17:35
3
$\begingroup$

Some additions to @civilstat's answer:

 Now that I have selected near-optimal hyperparameters, I can re-train the model on my whole dataset. There is no validation set though,

... which is why an additional data set to verify the final model's performance is needed whenever you need an estimate of the generalization performance of the final model.

Enter

  • nested cross validation

  • an additional held-out set independent of all data used for training the final model, including the test sets used for optimization.

  • Or, even better, a dedicated validation study for the model/method.

  • measuring performance during actual prediction use (this may be an ongoing QC to detect when the model becomes unsuitable e.g. due to drift)

  • In your particular example, random forest provides you automatically with out-of-bag estimates.
    (I know that the typical RF implementations do not offer grouped splitting, so the oob-estimate may be optimistically biased. But that affects its training as well...)

I have to wait for a real data point coming from the wild and be confident my model is the best possible.

While optimal and best possible [predictive performance] sound well, in practice, you typically need a model that predicts sufficiently well. Among those, robustness/ruggedness may be more important than the last little bit of performance on data obtained at the same time as the training data.

You judge your final model to be sufficiently good by verifying its performance with a suitable and independent data set.

Could it not be that I have overfitted when training on the whole dataset,

Of course. However, (repeated) k-fold CV can give you additional insight/warnings: if the (i⋅) k surrogate models differ a lot (may be difficult to judge for RF) or their predictions are unstable (differ a lot for the same input case; you can directly measure this for repeated k-fold CV), then your model training is not stable and you are overfitting.

and would not have been better to retain one (assuming there was a way to select) of the k surrogate models

  • Due to more training data, you'd expect the model trained on the whole data set to be equivalent or even better (slight pessimistic bias of CV) than any single of the $k$ (or $i \cdot k$) surrogate models. So on average, you'll do better to take the model trained on the whole data set.

  • Side note: which of the surrogate models do you keep? The first, the last, one randomly chosen? Why not another one?

    My answer here would be: if model training and optimization went well, they are equivalent, so it doesn't matter. If it does matter, better go for more rigorous training to stabilize the model(s).

    If you want to use the performance estimate obtained during cross validation as generalization error, you can not select based on these performance estimates*. Instead you take a pre-determined or randomly (by random number, as opposed to arbitrary picking!) chosen model.

  • Still, one may argue that a bird in the hand is worth two in the bush: for the surrogate models, you have actual, direct performance estimates. For the model trained on the whole data set, you haven't (except that for RF, you have the oob-estimate).

    Taking one of the surrogate models and using it as final model boils down to doing hold-out rather than cross validation. There are some situations where this may be preferrable.

    But consider: the performance estimate for each of the surrogate models comprises only $\frac{n}{k}$ tested cases. This estimate has variance (among other variance components) due to the finite number of tested cases. This variance will be $k$ times as much as the corresponding variance component for the model trained on the whole data set, where predictions from all $n$ cases are pooled into the CV performance estimate. OTOH, for the CV estimate applied to the model trained on the whole data, we expect a slight bias (due to larger training data size and learning curve) and an additional variance component due to model training instability, which we may approximate by the instability observed between the surrogate models.

    • if training instability dominates, you'd usually want to go back and develop a more stable model. However, if you need to do with what you have, your best bet would still not be taking one of the surrogate models but instead stabilizing the prediction by taking all of them into an ensemble. That is possible with cross validation just like with bootstrapping.

    • if the variance component due to number of tested cases dominates and training is stable, the model trained on the whole data set is still a better bet than any single surrogate model.

    • if training is stable, and sample size large, you may go with any (pre-determined/randomly chosen, no arbitrary picking) single of the surrogate models and its performance estimate and skip training on the whole data set.


* Why not select and use error estimate as generalization error

Due to the selection, the error estimate will become biased in a optimistic fashion:

When measuring prediction or generalization error, we observe and summarize the difference between the (surrogate) model's predictions and the true/reference value for the tested cases. Like any other measurement this is itself subject to error, in particular to variance (random error) due to the tested cases. (And the fewer tested cases, the more variance uncertainty, as usual.)

Now keeping aside the actual performance, we're more likely to select a model for which accidentally good performance was observed than a model for which accidentally bad performance was observed. Thus, the selection is biased towards models that happened to look better than they actually are. We are also more likely to select actually better models (which is what we want) but the extent of the selection bias depends on the size of the actual effect (better model) compared to the random error (variance, noise). Bias is expected to be higher for small numbers of tested cases, and for selecting from many models (multiple comparisons).

For the surrogate models of k-fold cross validation we actually have the assumption that they are equivalent, i.e. that we expect them to have (approximately) equal predictive performance. Thus, if we observe differences in the per-fold performance, we should by default expect them to be accidental. (With repeated k-fold cross validation, we can try to separate (in)stability-related variance from sample-size related variance more easily than with a single run.)

$\endgroup$
6
  • $\begingroup$ Great answer. Thanks especially for highlighting the problems with cherry-picking one of the surrogate models! $\endgroup$
    – civilstat
    Oct 10, 2023 at 15:16
  • $\begingroup$ @cbeleites unhappy with SX, thank you very much for the very instructive points. Would you please though expand on yiur statement "If you want to use the performance estimate obtained during cross validation as generalization error, you can not select based", not sure U grasp it in full, thanks $\endgroup$
    – Smerdjakov
    Oct 11, 2023 at 18:32
  • $\begingroup$ @cbeleites unhappy with SX, I was just reading your cited paper and thought about your statement, "This variance will be k times as much as the corresponding variance component for the model trained on the whole data set, where predictions from all n cases are pooled into the CV performance estimate". Would you please explsin how you estimate the fact the variance will be k time lower for the model trained on n points? I miss this completely, thanks do much $\endgroup$
    – Smerdjakov
    Feb 22 at 21:22
  • $\begingroup$ @Smerdjakov: so the performance estimate is subject to random error (variance). If you select based on such a measure, the variance creates a risk of selecting a model that does not have better generalization error, but "better looking" random error. In model optimization several factors can lead to this being a substantial risk. The result is that the performance estimate on which selection was based is an overoptimistic estimator of the selected model's performance. $\endgroup$ Feb 23 at 12:30
  • $\begingroup$ "variance will be k time lower for the model trained on n points" I think you may have misunderstood me. What I mean to say is: we have k surrogate models tested with n/k cases each, and these estimates have some variance due to the finite number of tested cases. When we look at the [usual] cross validation performance estimate that is pooled across the k folds, i.e., consists of estimates for all n cases - that estimate has lower variance by a factor k. (If you pool by averaging the k performance estimates, it's a clear variance of the mean conclusion. When pooling the individual predictions, $\endgroup$ Feb 23 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.