8
$\begingroup$

I feel like every question I've asked on CrossValidated has lead back to looking at the number of observations I have per variable. I understand that there are many rules of thumb out there depending on your field, your expected effect size, etc. These come back to recognizing that too few observations can lead to overfitting.

I understand how overfitting can be a problem when looking at a regression model with a single predictor. Having only two observations leads to a perfect answer, whereas using least squares to solve an overdetermined model leads to generalization. However, I have trouble making sense of how overfitting can still be a problem if you have 20 predictors and, say, 31 observations. It seems like you have mitigated the problem with 10 extra observations, but I suspect I am missing something in how least squares solves overdetermined systems.

What I assume follows if it is true that a system is overfitted is that the relationships between predictors explained by the betas also do not hold.

Finally, if overfitting is a problem by having not enough observations, can this be solved by using forward stepwise regression? Or is there a good possibility of missing significant predictors due to eventually reaching a point where there are too many predictors and not enough observations?

To recap:

  1. Can someone explain why the case where the number of predictors is $N-1$ where $N$ is decently large (say, $N > 10$) can still overfit? Why is it not that you need a minimum number of observations regardless of the number of predictors?

  2. Can overfitting be solved by using forward stepwise regression?

$\endgroup$
  • 3
    $\begingroup$ +1 I think this is a great question: it cogently challenges accepted wisdom, forcing us to think harder about why certain practices are correct and good. But, to narrow the scope a little, I would like to ask about your situation. (1) Are you certain the relationship between the dependent and all independent variables is linear? (2) Are you certain of your independent variables--that is, should they all be included or not? After all, you mention "overfitting": bear in mind, then, that with $p$ possible variables you have $2^{p+1}-1$ possible models to fit. That might require a lot of data! $\endgroup$ – whuber Apr 15 '13 at 13:02
  • 1
    $\begingroup$ (1) Yes, we are certain that the relationship is linear - we've been sure to do some trend analysis before starting the regression to see if we need any transformations. (2) We are not certain about our independent variables though. We are planning on performing some stepwise regression to learn more about the relationships that are present. $\endgroup$ – cryptic_star Apr 15 '13 at 13:37
  • 2
    $\begingroup$ Re (1) That's not "certain"--that's an estimate based on your data. You are, in effect, using your data for double duty: once to test whether the relationships appear linear, then again to fit the relationships. When you have only $N-1$ predictors, though, there is no way you can evaluate linearity. That already points to a need for some extra data. Re (2), perhaps your question ought to be "how is it even possible to choose wisely among $2^{p+1}-1$ models if I have fewer than $p + 2^{p+1}$ independent observations?" :-) $\endgroup$ – whuber Apr 15 '13 at 13:42
  • 2
    $\begingroup$ whuber, that last bit about choosing wisely among so many models when you have a fewer number of observations is the crux of the matter, and I'm still pondering it. ^_^ It still seems to me though that forward stepwise regression allows for getting closer to being able to make that choice with a smaller amount of data, although Scortchi's response below indicates that I may not have the most sound methodology with that thought. $\endgroup$ – cryptic_star Apr 15 '13 at 15:11
3
$\begingroup$

You miss the point with overfitting. It is not about the number of observations, but about in-sample and out-of-sample errors. The properly built model will have in-sample error approximately equal to out-of-sample error. Overfit model will have out-of-sample error larger than in-sample.

The purpose of modelling is not finding the best fit for your data, but finding the function which is able to predict the relationship, when you get new data not used for building the model. In such case the purpose is not to minimize the error, but minimize the difference between in- and out-of-sample errors.

Given the name of this page, I can't resist but link you to Cross-Validation.

$\endgroup$
  • 2
    $\begingroup$ Perhaps me missing the point of overfitting has to do with my perspective. Our end goal is not predictive, but rather to be able to describe relationships between predictors. However, I think I may have assumed that they both go together (a good predictive model also gives reliable relationships between predictors). Would you do anything different if your goal is to explore the nature of the relationships? $\endgroup$ – cryptic_star Apr 15 '13 at 12:13
  • $\begingroup$ Related, I come from a field where getting enough extra data to hold some out is a tough sell. I think I'm essentially looking to see if I can find a rough rule of thumb as to how much data I need per predictor to get my relationships between predictors. $\endgroup$ – cryptic_star Apr 15 '13 at 12:15
  • $\begingroup$ I see, the thing is that this perspective is no different. Finding relationship between some variables IS prediction. Why? Because what you want is to predict that when you get more data it will fit within the model you built! Otherwise the best fit would always be a 1000-power polynomial :) $\endgroup$ – sashkello Apr 15 '13 at 12:22
  • 2
    $\begingroup$ There is no sense in fitting a model without cross-validation. I understand that lack of data is frustrating, but this is all you can do - otherwise, as I said - you'll end up with N-power polynomial (order of the number of data points) fitting perfectly. Unfortunately I don't know what methods to use in such case (try searching for cross-validation with few data points), the sad truth is that most likely the best you can do is create models which give huge uncertainties. Not because they are bad but because this is the best you can do with the amount of data provided. $\endgroup$ – sashkello Apr 15 '13 at 12:25
1
$\begingroup$

And in response to your second question: Yes, forward selection can solve the problem, as can backward selection or stepwise; but these aren't the best ways of doing so. They reduce the variance in the predictions at the cost of introducing bias (by ignoring predictors). And there's not much principle behind them to give you confidence they're getting the balance right: a varying number of hypothesis tests at arbitrary significance levels. See Peter Flom's paper & the 'model-selection' tag here. (Though, to be fair, I've rarely found models fitted by such methods to be as bad as you might think - they do do roughly the right kind of thing.)

$\endgroup$
1
$\begingroup$

In relation to your second question, 'can overfitting be solved by using stepwise selection', I suggest that stepwise selection ignores relationships between variables too easily by focussing on the individual relationship of the predictor to the dependent variable. In addition the stepwise selection (I believe) means that the order in which the predictor variables are entered is influential.

I always examine the relationships between each of the dependent variables in a lot of detail prior to performing regression analysis, I also examine the effect that removal or addition has upon the remaining variables, and I check assumptions such as multicollinearity. Often, checking these assumptions enables decisions to be taken about the inclusion or removal of variables.

What I mean is that in order to determine which predictors to include, I spend a lot of time thinking about my research and examining the variables. I rarely just follow convention on the minimum number of observations (in fact I am much more strict than your example and usually ask for a minimum number of observations within subgroups) because my research is social in nature and often requires a lot of control variables.

My concern for your example is that unless your cohort are very similar (low variance amongst all variables), you won't be able to accurately observe relationships for uncommon subgroups because there aren't enough data to support these throughout the model, although I should add that I do not know what your independent variables are. For example if I have a small overall sample, and I am interested in looking at a gender effect, but I only have a small number of women observed then I would question whether gender should be included.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.