# How to detect when a regression model is over-fit?

When you are the one doing the work, being aware of what you are doing you develop a sense of when you have over-fit the model. For one thing, you can track the trend or deterioration in the Adjusted R Square of the model. You can also track a similar deterioration in the p values of the regression coefficients of the main variables.

But, when you just read someone else study and you have no insight as to their own internal model development process how can you clearly detect if a model is over-fit or not.

-
Just to throw a couple of ideas on the subject, if the study discloses standard regression statistics you could focus on the t stats and p values of the coefficients. If the RSquare of the model is high; but, one or more of the variables have a t stat < 2.0; this could be a red flag. Also, if the sign of the coefficients on some of the variables defy logic that is probably another red flag. If the study does not disclose a hold out period for the model, that may be another red flag. Hopefully, you will have other and better ideas. –  Gaetan Lion Sep 20 '10 at 23:12
One way is to see how the model performs on other (but similar) data. –  Shane Sep 20 '10 at 23:36

Cross validation and regularization are fairly common techniques to prevent overfitting. For a quick take, I'd recommend Andrew Moore's tutorial slides on the use of cross-validation -- pay particular attention to the caveats. For more detail, definitely read chapters 3 and 7 of EOSL, which cover the topic and associated matter in good depth.

-
Wow, thanks Andrew Moore's tutorial on cross-validation is world class. –  Gaetan Lion Sep 21 '10 at 4:08

When I'm fitting a model myself I generally use information criteria during the fitting process, such as AIC or BIC, or alternatively Likelihood-ratio tests for models fit based on maximum likelihood or F-test for models fit based on least squares.

All are conceptually similar in that they penalise additional parameters. They set a threshold of "additional explanatory power" for each new parameter added to a model. They are all a form of regularisation.

For others' models I look at the methods section to see if such techniques are used and also use rules of thumb, such as the number of observations per parameter - if there are around 5 (or fewer) observations per parameter I start to wonder.

Always remember that a variable need need not be "significant" in a model to be important. I may be a confounder and should be included on that basis if your goal is to estimate the effect of other variables.

-
Thanks for the links to the AIC and BIC tests. Do they add much value vs Adjusted R Square that does a similar thing by penalizing models for adding variables? –  Gaetan Lion Sep 21 '10 at 4:10
@Gaeten, Adjusted R-squared will increase when an F-test of the before vs after model is significant, so they are equivalent, except normally calculating an adjusted R-squared doesn't return a p-value. –  Thylacoleo Sep 21 '10 at 7:33
@Gaeten - AIC & BIC are more general than F-tests and adjusted R-squared which are usually limited to models fit by least squares. AIC & BIC can be used for any ,model where the likelihood can be calculated and degrees of freedom can be known (or estimated). –  Thylacoleo Sep 21 '10 at 7:44
Testing a set of variables is not a form of regularization (shrinkage). And testing gives one the temptation to remove variables, which has nothing to do with reducing overfitting. –  Frank Harrell Apr 26 '11 at 2:47

I would suggest that this is a problem with how the results are reported. Not to "beat the Bayesian drum" but approaching model uncertainty from a Bayesian perspective as an inference problem would greatly help here. And it doesn't have to be a big change either. If the report simply contained the probability that the model is true this would be very helpful. This is an easy quantity to approximate using BIC. Call the BIC for the mth model $BIC_{m}$. Then the probability that mth model is the "true" model, given that $M$ models were fit (and that one of the models is true) is given by:

$$P(\text{model m is true}|\text{one of the M models is true})\approx\frac{w_{m}\exp\left(-\frac{1}{2}BIC_{m}\right)}{\sum_{j=1}^{M}w_{j}\exp\left(-\frac{1}{2}BIC_{j}\right)}$$ $$=\frac{1}{1+\sum_{j\neq m}^{M}\frac{w_{j}}{w_{m}}\exp\left(-\frac{1}{2}(BIC_{j}-BIC_{m})\right)}$$

Where $w_{j}$ is proportional to the prior probability for the jth model. Note that this includes a "penalty" for trying to many models - and the penalty depends on how well the other models fit the data. Usually you will set $w_{j}=1$, however, you may have some "theoretical" models within your class that you would expect to be better prior to seeing any data.

Now if somebody else doesn't report all the BIC's from all the models, then I would attempt to infer the above quantity from what you have been given. Suppose you are given the BIC from the model - note that BIC is calculable from the mean square error of the regression model, so you can always get BIC for the reported model. Now if we take the basic premise that the final model was chosen from the smallest BIC then we have $BIC_{final}<BIC_{j}$. Now, suppose you were told that "forward" or "forward stepwise" model selection was used, starting from the intercept using $p$ potential variables. If the final model is of dimension $d$, then the procedure must have tried at least $$M\geq 1+p+(p-1)+\dots+(p-d+1)=1+\frac{p(p-1)-(p-d)(p-d-1)}{2}$$

different models (exact for forward selection), If the backwards selection was used, then we know at least $$M\geq 1+p+(p-1)+\dots+(d+1)=1+\frac{p(p-1)-d(d-1)}{2}$$

Models were tried (the +1 comes from the null model or the full model). Now we could try an be more specific, but these are "minimal" parameters which a standard model selection must satisfy. We could specify a probability model for the number of models tried $M$ and the sizes of the $BIC_{j}$ - but simply plugging in some values may be useful here anyway. For example suppose that all the BICs were $\lambda$ bigger than the one of the model chosen so that $BIC_{m}=BIC_{j}-\lambda$, then the probability becomes:

$$\frac{1}{1+(M-1)\exp\left(-\frac{\lambda}{2}\right)}$$

So what this means is that unless $\lambda$ is large or $M$ is small, the probability will be small also. From an "over-fitting" perspective, this would occur when the BIC for the bigger model is not much bigger than the BIC for the smaller model - a non-neglible term appears in the denominator. Plugging in the backward selection formula for $M$ we get:

$$\frac{1}{1+\frac{p(p-1)-d(d-1)}{2}\exp\left(-\frac{\lambda}{2}\right)}$$

Now suppose we invert the problem. say $p=50$ and the backward selection gave $d=20$ variables, what would $\lambda$ have to be to make the probability of the model greater than some value $P_{0}$? we have

$$\lambda > -2 log\left(\frac{2(1-P_{0})}{P_{0}[p(p-1)-d(d-1)]}\right)$$

Setting $P_{0}=0.9$ we get $\lambda > 18.28$ - so BIC of the winning model has to win by a lot for the model to be certain.

-