Explanation of minimum observations for multiple regression

Question

I feel like every question I've asked on CrossValidated has lead back to looking at the number of observations I have per variable. I understand that there are many rules of thumb out there depending on your field, your expected effect size, etc. These come back to recognizing that too few observations can lead to overfitting.

I understand how overfitting can be a problem when looking at a regression model with a single predictor. Having only two observations leads to a perfect answer, whereas using least squares to solve an overdetermined model leads to generalization. However, I have trouble making sense of how overfitting can still be a problem if you have 20 predictors and, say, 31 observations. It seems like you have mitigated the problem with 10 extra observations, but I suspect I am missing something in how least squares solves overdetermined systems.

What I assume follows if it is true that a system is overfitted is that the relationships between predictors explained by the betas also do not hold.

Finally, if overfitting is a problem by having not enough observations, can this be solved by using forward stepwise regression? Or is there a good possibility of missing significant predictors due to eventually reaching a point where there are too many predictors and not enough observations?

To recap:

1. Can someone explain why the case where the number of predictors is $N-1$ where $N$ is decently large (say, $N > 10$) can still overfit? Why is it not that you need a minimum number of observations regardless of the number of predictors?

2. Can overfitting be solved by using forward stepwise regression?

Answer 1

You miss the point with overfitting. It is not about the number of observations, but about in-sample and out-of-sample errors. The properly built model will have in-sample error approximately equal to out-of-sample error. Overfit model will have out-of-sample error larger than in-sample.

The purpose of modelling is not finding the best fit for your data, but finding the function which is able to predict the relationship, when you get new data not used for building the model. In such case the purpose is not to minimize the error, but minimize the difference between in- and out-of-sample errors.

Given the name of this page, I can't resist but link you to Cross-Validation.

Answer 2

And in response to your second question: Yes, forward selection can solve the problem, as can backward selection or stepwise; but these aren't the best ways of doing so. They reduce the variance in the predictions at the cost of introducing bias (by ignoring predictors). And there's not much principle behind them to give you confidence they're getting the balance right: a varying number of hypothesis tests at arbitrary significance levels. See Peter Flom's paper & the 'model-selection' tag here. (Though, to be fair, I've rarely found models fitted by such methods to be as bad as you might think - they do do roughly the right kind of thing.)

Answer 3

In relation to your second question, 'can overfitting be solved by using stepwise selection', I suggest that stepwise selection ignores relationships between variables too easily by focussing on the individual relationship of the predictor to the dependent variable. In addition the stepwise selection (I believe) means that the order in which the predictor variables are entered is influential.

I always examine the relationships between each of the dependent variables in a lot of detail prior to performing regression analysis, I also examine the effect that removal or addition has upon the remaining variables, and I check assumptions such as multicollinearity. Often, checking these assumptions enables decisions to be taken about the inclusion or removal of variables.

What I mean is that in order to determine which predictors to include, I spend a lot of time thinking about my research and examining the variables. I rarely just follow convention on the minimum number of observations (in fact I am much more strict than your example and usually ask for a minimum number of observations within subgroups) because my research is social in nature and often requires a lot of control variables.

My concern for your example is that unless your cohort are very similar (low variance amongst all variables), you won't be able to accurately observe relationships for uncommon subgroups because there aren't enough data to support these throughout the model, although I should add that I do not know what your independent variables are. For example if I have a small overall sample, and I am interested in looking at a gender effect, but I only have a small number of women observed then I would question whether gender should be included.