3
$\begingroup$

I want to do a logistic regression with R.

I have 18 continuous covariates and a sample consisting of 100 observations.

When I enter all of the covariates into the glm() model, none of them are significant, but the model predicts the outcome perfectly on the test data!

My questions are:

  1. Is the sample size large enough for running glm() with this many covariates?
  2. What might be other causes of the problem?
  3. How can I run such a model properly?
Improve this question
$\endgroup$
2
  • $\begingroup$ That's a lot of covariates - do you need to include all of them? Can you use a model selection procedure to narrow it down to a few important covariates? $\endgroup$
    – atrichornis
    Commented Jul 31, 2013 at 9:00
  • 4
    $\begingroup$ It's not surprising that your model predicts perfectly since you have so many degrees of freedom in your model (overfitting). You could use penalized regression (e.g., package glmnet). $\endgroup$
    – Roland
    Commented Jul 31, 2013 at 9:11

2 Answers 2

Reset to default
5
$\begingroup$

You did not state the number of events and non-events. A rough rule of thumb is that to use ordinary maximum likelihood estimation (i.e., without shrinkage - penalization) requires 15 times the number of events and the number of non-events as the number of candidate predictors. You are far from having an adequate sample size in your case even if $Y$ is split 50-50. I suggest doing data reduction (masked to $Y$, e.g. variable clustering, principal components, or redundancy analysis) or fitting the full list of variables and solving for the amount of shrinkage needed to yield a reliable model. You can see case studies of these methods in my Handouts at http://biostat.mc.vanderbilt.edu/CourseBios330.

Ordinary unpenalized variable selection methods in no way deal with this problem correctly.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Can you please provide a citation for the 15 estimate? Once upon a time someone I was working with tried and I found one that recommended 10 based on simulation (and a correction for unequal proportions and number of predictors). $\endgroup$
    – russellpierce
    Commented Jul 31, 2013 at 23:48
  • $\begingroup$ See the pdf attachment at the bottom of biostat.mc.vanderbilt.edu/FrankHarrell. $\endgroup$
    – Frank Harrell
    Commented Aug 1, 2013 at 11:18
  • $\begingroup$ Yes, Y is split 50-50. $\endgroup$
    – Mahmoud
    Commented Aug 4, 2013 at 5:54
1
$\begingroup$

It would help if you could provide some additional information in response to the comments and @Frank Harrell's point regarding how many successes and failures you have.

My first guess would be that you have some multicollinearity, that is, your continuous covariates are correlated with each other. The effect of this is that, while your betas estimates are still potentially unbiased, the standard errors will be inflated. That means that they will be less 'significant', but may still do a good job of predicting the response (note @Roland's good point about overfitting, however).

Given your N and the number of covariates you have, you may want to watch out for quasi-separation as well.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.