2
$\begingroup$

I want to do a logistic regression with R.

I have 18 continuous covariates and a sample consisting of 100 observations.

When I enter all of the covariates into the glm() model, none of them are significant, but the model predicts the outcome perfectly on the test data!

My questions are:

  1. Is the sample size large enough for running glm() with this many covariates?
  2. What might be other causes of the problem?
  3. How can I run such a model properly?
$\endgroup$
  • $\begingroup$ That's a lot of covariates - do you need to include all of them? Can you use a model selection procedure to narrow it down to a few important covariates? $\endgroup$ – atrichornis Jul 31 '13 at 9:00
  • 3
    $\begingroup$ It's not surprising that your model predicts perfectly since you have so many degrees of freedom in your model (overfitting). You could use penalized regression (e.g., package glmnet). $\endgroup$ – Roland Jul 31 '13 at 9:11
5
$\begingroup$

You did not state the number of events and non-events. A rough rule of thumb is that to use ordinary maximum likelihood estimation (i.e., without shrinkage - penalization) requires 15 times the number of events and the number of non-events as the number of candidate predictors. You are far from having an adequate sample size in your case even if $Y$ is split 50-50. I suggest doing data reduction (masked to $Y$, e.g. variable clustering, principal components, or redundancy analysis) or fitting the full list of variables and solving for the amount of shrinkage needed to yield a reliable model. You can see case studies of these methods in my Handouts at http://biostat.mc.vanderbilt.edu/CourseBios330.

Ordinary unpenalized variable selection methods in no way deal with this problem correctly.

$\endgroup$
  • $\begingroup$ Can you please provide a citation for the 15 estimate? Once upon a time someone I was working with tried and I found one that recommended 10 based on simulation (and a correction for unequal proportions and number of predictors). $\endgroup$ – russellpierce Jul 31 '13 at 23:48
  • $\begingroup$ See the pdf attachment at the bottom of biostat.mc.vanderbilt.edu/FrankHarrell. $\endgroup$ – Frank Harrell Aug 1 '13 at 11:18
  • $\begingroup$ Yes, Y is split 50-50. $\endgroup$ – Mahmoud Aug 4 '13 at 5:54
1
$\begingroup$

It would help if you could provide some additional information in response to the comments and @Frank Harrell's point regarding how many successes and failures you have.

My first guess would be that you have some multicollinearity, that is, your continuous covariates are correlated with each other. The effect of this is that, while your betas estimates are still potentially unbiased, the standard errors will be inflated. That means that they will be less 'significant', but may still do a good job of predicting the response (note @Roland's good point about overfitting, however).

Given your N and the number of covariates you have, you may want to watch out for quasi-separation as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.