9
$\begingroup$

Is there a measure in logistic regression that maybe penalizes you for having too many independent variables like in multiple regression with the adjusted R squared?

That is, does having too many independent variables in a logistic regression hurt the model?

What about dummy variables? Can you have too many of those to the point of unpredictability?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ The people at Google run logistic regressions with hundreds of thousands of predictors (mostly dummies) to decide what ads to show. (Obviously they have some fairly large samples). $\endgroup$
    – Jeremy Miles
    Dec 13, 2013 at 5:39

3 Answers 3

Reset to default
11
$\begingroup$

For the typical low signal:noise ratio we see in most problems, a common rule of thumb is that you need about 15 times as many events and 15 times as many non-events as there are parameters that you entertain putting into the model. The rationale for that "rule" is that it results in a model performance metric that is likely to be as good or as bad in new data as it appears to be in the training data. But you need 96 observations just to estimate the intercept so that the overall predicted risk is within a $\pm 0.1$ margin of error of the true risk with 0.95 confidence.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ So if you have 5 independent binary variables you need to have at least 75 true and 75 false observations for each one? $\endgroup$
    – Micro
    Dec 12, 2013 at 14:46
  • $\begingroup$ If you have 5 candidate variables, they are all binary, and you don't posit any interactions, you would need at least 75 events and non-events total. But since you need 96 observations to estimate the intercept reliably, somehow add that into the rule of thumb. $\endgroup$
    – Frank Harrell
    Dec 12, 2013 at 16:14
  • $\begingroup$ @FrankHarrell any academic reference for this? $\endgroup$
    – Learner
    Mar 14, 2017 at 8:27
  • 1
    $\begingroup$ My course notes at biostat.mc.vanderbilt.edu/rms $\endgroup$
    – Frank Harrell
    Mar 14, 2017 at 14:57
3
$\begingroup$

Having too many parameters compared to observations may lead to overfitting. Various adjustments or measures can be used to correct for this. AIC for example accounts for both the number of variables and the number of observations in your dataset and is probably most often used. AIC itself doesn't adjust the model, but serves as a tool to select the best model if you construct multiple ones. It's basically a tradeoff between residual error and model complexity.

You can furthermore take a look at other "information criteria" or more advanced techniques like crossvalidation, penalized logistic regression ("penalized" package in R), ...

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Not clear on how AIC adjusts parameter estimates for overfitting. $\endgroup$
    – Frank Harrell
    Dec 11, 2013 at 22:18
  • $\begingroup$ Indeed, post edited. $\endgroup$
    – Matt.135
    Dec 12, 2013 at 9:14
3
$\begingroup$

If the number of independent variables is not very large, you can just do “all subsets” regression in which all possible models are fit. The model the model with the highest F statistic or proportion of explained variation (PVE) (note: the concept was established with linear regression but can be applied to logistic regression as well) is selected. But this often results that we will choose the full model. So we need to penalize models with many variables that don’t fit much better than models with fewer variables with Akaike Information Criterion (AIC). Lower AIC values usually indicate a better model that we will finally select.

If the number of independent variables is large. The strategy is, select the best model with only one variable, then select another variable so that the best model with two variables is obtained, then select the 3rd variable...so on and so forth. The selection stops once AIC increases. Usually the complexity is around O(n^2) rather than O(2^n) in all subsets regression.

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ The performance of this approach is incredibly bad and explains why a good deal of published research is non-reproducible. $\endgroup$
    – Frank Harrell
    Dec 15, 2013 at 15:26
  • $\begingroup$ Do you mean it is computational inefficient, or actually the best variable combination does not necessarily have the least AIC? $\endgroup$
    – lennon310
    Dec 15, 2013 at 15:50
  • 2
    $\begingroup$ I mean that variable selection creates overfitting, huge biases, and destroys all aspects of statistical inference. $\endgroup$
    – Frank Harrell
    Dec 15, 2013 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.