I am using logit models to predict whether or not children are unhealthy (binary indictor). Many of my models have statistically significant relationships between predictors of interest (ie, if the child’s household has improved water/sanitation) and the outcome. However, none of my models outperform a model which just uses demographic information (age, region, urban/rural, sex). Furthermore, the models are pretty terrible at predicting unhealthy children (only about ~15% true positives correctly predicted, which are about 23% of the data). Do these two facts render the statistically significant relationships between the predictors and outcome of interest meaningless? How do I explain this to my (hopefully future) readers? Thanks,


Here are the set of results from my model -- I've masked the actual variables names because I cannot share them publicly.

variable    controls    proximal variables
(Intercept) 0.212(0.255)    0.172(0.296)
variable 1                  1.038(0.09)
variable 2                  0.721(0.098)**
variable 3                  0.986(0.095)
variable 4                  0.810(0.104)**
variable 5                  0.744(0.109)**
variable 6                  0.981(0.15)
variable 7                  1.317(0.051)**
controls 1  0.769(0.087)**  0.763(0.087)**
controls 2  1.078(0.015)**  1.087(0.015)**
controls 3  0.666(0.098)**  0.838(0.11)
controls 4  0.958(0.122)    1.129(0.127)
controls 5  0.706(0.144)**  0.750(0.146)**
controls 6  1.012(0.086)    0.998(0.089)
controls 7  0.957(0.141)    1.010(0.142)
controls 8  0.694(0.173)**  0.726(0.18)*
controls 9  1.895(0.065)**  1.575(0.072)**
controls 10 1.353(0.085)**  1.279(0.095)**
controls 11 0.907(0.083)    0.949(0.095)
controls 12 0.988(0.135)    0.920(0.145)
Log Likelihood  -3417.879   -3356.028
AIC                   6863.758  6754.056
BIC                   6958.156  6895.653
PCP                     0.702   0.697

I am a little off topic because I just offer you an other example of this kind of behavior, but hopefully this is extreme enough to help you understanding it.

So my point is that when a factor associated with a trait is frequent, it can be a terrible predictor, even if the association is strong. Here is R code for an hypothetic case/control study. First, generate y and x:

y <- as.factor( c( rep(0,100), rep(1,100) ) )
x <- as.factor( c( sample(0:1, 100, replace=TRUE),
                   sample(0:1, 100, replace=TRUE, prob=c(1,99) )))

Then run a logistic regression:

> summary(glm(y ~ x, family=binomial(link=logit)))

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-1.4916  -0.2747   0.3091   0.8929   2.5674

            Estimate Std. Error z value Pr(>|z|)
(Intercept)  -3.2581     0.7203  -4.523 6.08e-06 ***
x1            3.9719     0.7415   5.357 8.48e-08 ***

The $\beta_1$ value is huge, isn’t it? And the $p$-value is tiny. Don’t pay attention to the intercept, this is a case/control study.

As huge as $\beta_1$ is, $x_1$ is a terrible predictor: of course, 99% out of the cases ($y=1$) have $x=1$, but still 50% of controls have $x=1$. So if you use $x=1$ to predict $y=1$, you will misclassify 50% of the controls (and if the trait $y=1$ is rare, near to 50% of the population).

You can modify this example to get the reverse behavior (correctly classifying controls but missing many cases), etc. You will even get a terrible behavior is all categories like this :

y <- as.factor( c( rep(0,100), rep(1,100) ) ) 
x <- as.factor( c( sample(0:1, 100, replace=TRUE, prob=c(7,3)),
                   sample(0:1, 100, replace=TRUE, prob=c(3,7) )))

Let's go:

> summary(glm(y ~ x, family=binomial(link=logit)))

Deviance Residuals: 
      Min        1Q    Median        3Q       Max  
-1.54527  -0.85630  -0.00329   0.84972   1.53697  

            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -0.8145     0.2157  -3.775  0.00016 ***
x1            1.6474     0.3072   5.363  8.2e-08 ***

The $\beta$ is still quite large, but you can’t classify with this variable...

PS See also this paper (in Genetics)

| cite | improve this answer | |
  • 2
    $\begingroup$ That's a great example -- if you (or someone else) could discuss the implications of this, that would be great. Is discussing the association still valid, even if it is a poor predictor? How have you framed this in the past? $\endgroup$ – mike Dec 10 '12 at 1:02
  • $\begingroup$ Oh, yes, the association is genuine, and it is legit to discuss it. If one doesn't believe in randomness, and think that's everything is deterministic, randomness being just a way to account for hidden variables, one should conclude that there are other variables yet unmeasured. $\endgroup$ – Elvis Dec 10 '12 at 6:01
  • $\begingroup$ However it is still possible that the residual randomness can't be reduced much further. Or at least, in your case, not by observing sociological variables. Did you try using demographic variables and your variables together? $\endgroup$ – Elvis Dec 10 '12 at 6:10
  • $\begingroup$ Yes -- that is what is happening in the second column of the table -- thanks for the confidence. $\endgroup$ – mike Dec 10 '12 at 6:14
  • 1
    $\begingroup$ I mean, you should try to see how your variables improve prediction based solely on demographic information. Maybe, a few interaction terms could be added. $\endgroup$ – Elvis Dec 10 '12 at 8:47

You're not alone; lots of us feel embarrassed trying to explain to our audiences that, even with stat. sig. results, we still can't classify cases correctly very often. But remember that classification is partially under your control. You don't have to accept your software's probable default of 0.5 as a threshold for the predicted probability that will get a case classified as a predicted "1." Maybe in your context you want to set that threshold (cut point) much lower. Studying an ROC curve will help you decide what your options are and how you want to handle the tradeoff between sensitivity (identifying a large enough fraction of the true positives) and lack of specificity (ending up with too many false positives).

| cite | improve this answer | |

Statistical significance is not all that significant, in many cases. Effect sizes are more meaningful.

One way the results you describe can happen is if there is a large sample. This makes even small effects significant.

Since you are looking at logistic regression models, I would examine the odds ratios and see if they are interestingly large.

| cite | improve this answer | |
  • $\begingroup$ Thanks Peter, that seems reassuring, but how is it that I can reconcile the fact (to others) that the effect sizes are large and significant, but the models themselves are poorly performing? I want to report the odds ratios, but the model performance seems to undercut the believability of my analysis. $\endgroup$ – mike Dec 9 '12 at 23:19
  • $\begingroup$ If the effect sizes are large and significant, the model must perform well. That's sort of what a large effect size means. $\endgroup$ – Peter Flom Dec 9 '12 at 23:21
  • $\begingroup$ Maybe I'm misunderstanding what you mean by effect size: the betas are large and statisically significant, but when comparing actual versus predicted, the model performs pretty poorly. $\endgroup$ – mike Dec 9 '12 at 23:35
  • $\begingroup$ Something strange is going on, then. Can you post your results? $\endgroup$ – Peter Flom Dec 9 '12 at 23:47
  • 1
    $\begingroup$ Nothing strange to me. Large odds ratios and poor specificity, this happens a lot in my field (genetics). $\endgroup$ – Elvis Dec 9 '12 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.