I feel like every question I've asked on CrossValidated has lead back to looking at the number of observations I have per variable. I understand that there are many rules of thumb out there depending on your field, your expected effect size, etc. These come back to recognizing that too few observations can lead to overfitting.

I understand how overfitting can be a problem when looking at a regression model with a single predictor. Having only two observations leads to a perfect answer, whereas using least squares to solve an overdetermined model leads to generalization. However, I have trouble making sense of how overfitting can still be a problem if you have 20 predictors and, say, 31 observations. It seems like you have mitigated the problem with 10 extra observations, but I suspect I am missing something in how least squares solves overdetermined systems.

What I assume follows if it is true that a system is overfitted is that the relationships between predictors explained by the betas also do not hold.

Finally, if overfitting is a problem by having not enough observations, can this be solved by using forward stepwise regression? Or is there a good possibility of missing significant predictors due to eventually reaching a point where there are too many predictors and not enough observations?

To recap:

1. Can someone explain why the case where the number of predictors is N-1 where N is decently large (say, N > 10) can still overfit? Why is it not that you need a minimum number of observations regardless of the number of predictors?

2. Can overfitting be solved by using forward stepwise regression?

I feel like every question I've asked on CrossValidated has lead back to looking at the number of observations I have per variable. I understand that there are many rules of thumb out there depending on your field, your expected effect size, etc. These come back to recognizing that too few observations can lead to overfitting.

I understand how overfitting can be a problem when looking at a regression model with a single predictor. Having only two observations leads to a perfect answer, whereas using least squares to solve an overdetermined model leads to generalization. However, I have trouble making sense of how overfitting can still be a problem if you have 20 predictors and, say, 31 observations. It seems like you have mitigated the problem with 10 extra observations, but I suspect I am missing something in how least squares solves overdetermined systems.

What I assume follows if it is true that a system is overfitted is that the relationships between predictors explained by the betas also do not hold.

Finally, if overfitting is a problem by having not enough observations, can this be solved by using forward stepwise regression? Or is there a good possibility of missing significant predictors due to eventually reaching a point where there are too many predictors and not enough observations?

To recap:

1. Can someone explain why the case where the number of predictors is $N-1$ where $N$ is decently large (say, $N > 10$) can still overfit? Why is it not that you need a minimum number of observations regardless of the number of predictors?

2. Can overfitting be solved by using forward stepwise regression?