# Maximum number of independent variables in Logistic Regression

Is there a measure in logistic regression that maybe penalizes you for having too many independent variables like in multiple regression with the adjusted R squared?

That is, does having too many independent variables in a logistic regression hurt the model?

What about dummy variables? Can you have too many of those to the point of unpredictability?

• The people at Google run logistic regressions with hundreds of thousands of predictors (mostly dummies) to decide what ads to show. (Obviously they have some fairly large samples). Dec 13, 2013 at 5:39

For the typical low signal:noise ratio we see in most problems, a common rule of thumb is that you need about 15 times as many events and 15 times as many non-events as there are parameters that you entertain putting into the model. The rationale for that "rule" is that it results in a model performance metric that is likely to be as good or as bad in new data as it appears to be in the training data. But you need 96 observations just to estimate the intercept so that the overall predicted risk is within a $\pm 0.1$ margin of error of the true risk with 0.95 confidence.

• So if you have 5 independent binary variables you need to have at least 75 true and 75 false observations for each one? Dec 12, 2013 at 14:46
• If you have 5 candidate variables, they are all binary, and you don't posit any interactions, you would need at least 75 events and non-events total. But since you need 96 observations to estimate the intercept reliably, somehow add that into the rule of thumb. Dec 12, 2013 at 16:14
• @FrankHarrell any academic reference for this? Mar 14, 2017 at 8:27
• My course notes at biostat.mc.vanderbilt.edu/rms Mar 14, 2017 at 14:57

Having too many parameters compared to observations may lead to overfitting. Various adjustments or measures can be used to correct for this. AIC for example accounts for both the number of variables and the number of observations in your dataset and is probably most often used. AIC itself doesn't adjust the model, but serves as a tool to select the best model if you construct multiple ones. It's basically a tradeoff between residual error and model complexity.

You can furthermore take a look at other "information criteria" or more advanced techniques like crossvalidation, penalized logistic regression ("penalized" package in R), ...

• Not clear on how AIC adjusts parameter estimates for overfitting. Dec 11, 2013 at 22:18
• Indeed, post edited. Dec 12, 2013 at 9:14

If the number of independent variables is not very large, you can just do “all subsets” regression in which all possible models are fit. The model the model with the highest F statistic or proportion of explained variation (PVE) (note: the concept was established with linear regression but can be applied to logistic regression as well) is selected. But this often results that we will choose the full model. So we need to penalize models with many variables that don’t ﬁt much better than models with fewer variables with Akaike Information Criterion (AIC). Lower AIC values usually indicate a better model that we will finally select.

If the number of independent variables is large. The strategy is, select the best model with only one variable, then select another variable so that the best model with two variables is obtained, then select the 3rd variable...so on and so forth. The selection stops once AIC increases. Usually the complexity is around O(n^2) rather than O(2^n) in all subsets regression.

• The performance of this approach is incredibly bad and explains why a good deal of published research is non-reproducible. Dec 15, 2013 at 15:26
• Do you mean it is computational inefficient, or actually the best variable combination does not necessarily have the least AIC? Dec 15, 2013 at 15:50
• I mean that variable selection creates overfitting, huge biases, and destroys all aspects of statistical inference. Dec 15, 2013 at 20:57