0
$\begingroup$

I'm using logistic regression to look for association between independent variables and outcome (i.e. not to create a classifier). I have many variables, and a sample size of 391 (with 58 y=1 and the rest all y=0). By the rule of 10s, I am limited to 5 independent variables.

Is it possible to test associations by creating two models, one on a subset independent variables (IV1, IV2, IV3, IV4, and IV5) and the other on the remaining ones (IV6, IV7, IV8)? I would then test each model separately for over-fitting, etc.

My result could, theoretically, be IV3 has p-value 0.002 and IV8 has p-value 0.01 and I would conclude (assuming goodness-of-fit tests are good) that there is an association between outcome and IV3 as well as IV8. Does that make sense?

$\endgroup$
  • $\begingroup$ Sorry, I misread your original post. How many predictors do you have? I don't think there's any reason to "partition" the set of predictors. You could simply look at all $5$ variable combinations (no need to look at the model with the remaining $p - 5$ predictors) and choose the set that yield the best model, where best could be defined using AIC, the so-called "$c$-statistic" or some other metric. Also, I'm not familiar with a rule of $10$ and don't see why you need a model with $5$ predictors. $\endgroup$ – dsaxton Feb 22 '16 at 3:29
  • $\begingroup$ Regularization methods like lasso/ridge/elastic neg will let you fit regressions even in the case of having more features than examples. $\endgroup$ – Sycorax says Reinstate Monica Feb 22 '16 at 6:02
  • $\begingroup$ I have 4 independent variables. 1 of them is categorical (non-ordinal) with 5 categories, so I actually end up with 8. I am not building a predictor - merely looking for associations between each of those and the outcome. I was thinking of building two models, one with the 5 categories, and one with the remaining 3 other parameters. $\endgroup$ – middleAgedProgrammer Feb 23 '16 at 1:57
  • $\begingroup$ The rule of 10 is a limit on the number of predictors based on the frequency of the less common outcome. More info here en.wikipedia.org/wiki/One_in_ten_rule $\endgroup$ – middleAgedProgrammer Feb 23 '16 at 2:08
  • $\begingroup$ Your 5-level categorical variable only counts as 4 IVs in the regression. Think about a binary true/false predictor: that's only 1 IV despite its 2 possible values. Then add 3 more categories. $\endgroup$ – EdM Feb 23 '16 at 14:52
2
$\begingroup$

Your approach gives up one of the advantages of multiple regression: accounting for the combined influences of all the predictors at once. It's thus effectively throwing away information, which is seldom useful.

One way to deal with too many predictors is to use subject-matter knowledge or the observed relations among the predictors (not considering the outcomes) to combine some related predictors into a combined individual predictor.

Another way was suggested in the comment by @user777: use LASSO, ridge regression, or elastic net, which impose a penalty on regression coefficients that guards against overfitting. (The rule of thumb of 10 events per variable was based on non-penalized analyses.) These methods provide principled ways to build models even if you have more predictor variables than cases.

Note that the best-subset suggestion in another comment doesn't get around the overfitting issue, and the variables selected would be highly dependent on your particular data sample. Try repeating best-subset analysis on multiple bootstrap samples to see the problems.

If your interest is prediction and you only have 8 predictor variables, ridge regression will probably work well on your data.

Added in response to comments:

The paper linked from a comment, on assessing multivariable logistic regression models, rightly emphasizes the proper selection of predictor variables as a major criterion. Using subject-matter knowledge for selecting or combining variables should be a top priority. You might, for example, be able to combine categories in your categorical variable, or omit other predictors that have been shown in related studies not to be closely related to outcome.

That paper's sole focus on 10 events per variable to prevent overfitting, however, is inadequate in two ways. First, as a rule of thumb, 15 events per variable may be a better choice than 10. Second, not noted in that paper, methods like LASSO and ridge regression provide another well-established way to prevent overfitting, by shrinking the magnitudes of the coefficients to less than those that would be provided by standard logistic regression. See for example An Introduction to Statistical Learning for background on these and other approaches.

The idea to break your analysis into 2 parts (5-level categorical variable and then all other variables separately) doesn't really accomplish much. What you think you might gain from having about 15-20 events per variable in each of the 2 separate analyses would be lost by your need to correct for multiple hypothesis testing and your inability to take into account the levels of the categorical variable when evaluating the other variables (and vice-versa). And you would still need to evaluate overfitting.

With respect to "investigating associations" versus predictive modeling, consider what Frank Harrell has to say in Regression Modeling Strategies, second edition, page 3:

Thus when one develops a reasonable multivariable predictive model, hypothesis testing and estimation of effects are byproducts of the fitted model. So predictive modeling is often desirable even when prediction is not the main goal.

Harrell's rms package in R provides the tools you need to build, calibrate, and validate logistic models. His book linked above and associated class notes provide examples of ways to deal with too many variables. Try them out on your dataset.

$\endgroup$
  • $\begingroup$ For this problem, I'm not building a predictor; rather, I'd like to investigate associations between independent variables and a binary outcome. In my field, it is common to use logistic regression (e.g. onlinelibrary.wiley.com/doi/10.1111/j.1600-6143.2010.03141.x/… ) and since I'm not providing a second dataset for validation, I'd like to stick to established methods. My intuition was that if I could show associations in 2 separate models, yes I would lose info, but I would end up with a conservative estimate of the associations, which is good enough. Is this reasoning sound? $\endgroup$ – middleAgedProgrammer Feb 23 '16 at 2:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.