How to detect when a regression model is over-fit?

Question

When you are the one doing the work, being aware of what you are doing you develop a sense of when you have over-fit the model. For one thing, you can track the trend or deterioration in the Adjusted R Square of the model. You can also track a similar deterioration in the p values of the regression coefficients of the main variables.

But, when you just read someone else study and you have no insight as to their own internal model development process how can you clearly detect if a model is over-fit or not.

Answer 1

Cross validation and regularization are fairly common techniques to prevent overfitting. For a quick take, I'd recommend Andrew Moore's tutorial slides on the use of cross-validation -- pay particular attention to the caveats. For more detail, definitely read chapters 3 and 7 of EOSL, which cover the topic and associated matter in good depth.

Answer 2

When I'm fitting a model myself I generally use information criteria during the fitting process, such as AIC or BIC, or alternatively Likelihood-ratio tests for models fit based on maximum likelihood or F-test for models fit based on least squares.

All are conceptually similar in that they penalise additional parameters. They set a threshold of "additional explanatory power" for each new parameter added to a model. They are all a form of regularisation.

For others' models I look at the methods section to see if such techniques are used and also use rules of thumb, such as the number of observations per parameter - if there are around 5 (or fewer) observations per parameter I start to wonder.

Always remember that a variable need need not be "significant" in a model to be important. I may be a confounder and should be included on that basis if your goal is to estimate the effect of other variables.

Answer 3

I would suggest that this is a problem with how the results are reported. Not to "beat the Bayesian drum" but approaching model uncertainty from a Bayesian perspective as an inference problem would greatly help here. And it doesn't have to be a big change either. If the report simply contained the probability that the model is true this would be very helpful. This is an easy quantity to approximate using BIC. Call the BIC for the mth model $BIC_{m}$. Then the probability that mth model is the "true" model, given that $M$ models were fit (and that one of the models is true) is given by:

$$P(\text{model m is true}|\text{one of the M models is true})\approx\frac{w_{m}\exp\left(-\frac{1}{2}BIC_{m}\right)}{\sum_{j=1}^{M}w_{j}\exp\left(-\frac{1}{2}BIC_{j}\right)}$$ $$=\frac{1}{1+\sum_{j\neq m}^{M}\frac{w_{j}}{w_{m}}\exp\left(-\frac{1}{2}(BIC_{j}-BIC_{m})\right)}$$

Where $w_{j}$ is proportional to the prior probability for the jth model. Note that this includes a "penalty" for trying to many models - and the penalty depends on how well the other models fit the data. Usually you will set $w_{j}=1$, however, you may have some "theoretical" models within your class that you would expect to be better prior to seeing any data.

Now if somebody else doesn't report all the BIC's from all the models, then I would attempt to infer the above quantity from what you have been given. Suppose you are given the BIC from the model - note that BIC is calculable from the mean square error of the regression model, so you can always get BIC for the reported model. Now if we take the basic premise that the final model was chosen from the smallest BIC then we have $BIC_{final}<BIC_{j}$. Now, suppose you were told that "forward" or "forward stepwise" model selection was used, starting from the intercept using $p$ potential variables. If the final model is of dimension $d$, then the procedure must have tried at least $$M\geq 1+p+(p-1)+\dots+(p-d+1)=1+\frac{p(p-1)-(p-d)(p-d-1)}{2}$$

different models (exact for forward selection), If the backwards selection was used, then we know at least $$M\geq 1+p+(p-1)+\dots+(d+1)=1+\frac{p(p-1)-d(d-1)}{2}$$

Models were tried (the +1 comes from the null model or the full model). Now we could try an be more specific, but these are "minimal" parameters which a standard model selection must satisfy. We could specify a probability model for the number of models tried $M$ and the sizes of the $BIC_{j}$ - but simply plugging in some values may be useful here anyway. For example suppose that all the BICs were $\lambda$ bigger than the one of the model chosen so that $BIC_{m}=BIC_{j}-\lambda$, then the probability becomes:

$$\frac{1}{1+(M-1)\exp\left(-\frac{\lambda}{2}\right)}$$

So what this means is that unless $\lambda$ is large or $M$ is small, the probability will be small also. From an "over-fitting" perspective, this would occur when the BIC for the bigger model is not much bigger than the BIC for the smaller model - a non-neglible term appears in the denominator. Plugging in the backward selection formula for $M$ we get:

$$\frac{1}{1+\frac{p(p-1)-d(d-1)}{2}\exp\left(-\frac{\lambda}{2}\right)}$$

Now suppose we invert the problem. say $p=50$ and the backward selection gave $d=20$ variables, what would $\lambda$ have to be to make the probability of the model greater than some value $P_{0}$? we have

$$\lambda > -2 log\left(\frac{2(1-P_{0})}{P_{0}[p(p-1)-d(d-1)]}\right)$$

Setting $P_{0}=0.9$ we get $\lambda > 18.28$ - so BIC of the winning model has to win by a lot for the model to be certain.