Interpret statistically significant betas with poorly performing models

Question

I am using logit models to predict whether or not children are unhealthy (binary indictor). Many of my models have statistically significant relationships between predictors of interest (ie, if the child’s household has improved water/sanitation) and the outcome. However, none of my models outperform a model which just uses demographic information (age, region, urban/rural, sex). Furthermore, the models are pretty terrible at predicting unhealthy children (only about ~15% true positives correctly predicted, which are about 23% of the data). Do these two facts render the statistically significant relationships between the predictors and outcome of interest meaningless? How do I explain this to my (hopefully future) readers? Thanks,

EDIT:

Here are the set of results from my model -- I've masked the actual variables names because I cannot share them publicly.

variable    controls    proximal variables
(Intercept) 0.212(0.255)    0.172(0.296)
variable 1                  1.038(0.09)
variable 2                  0.721(0.098)**
variable 3                  0.986(0.095)
variable 4                  0.810(0.104)**
variable 5                  0.744(0.109)**
variable 6                  0.981(0.15)
variable 7                  1.317(0.051)**
controls 1  0.769(0.087)**  0.763(0.087)**
controls 2  1.078(0.015)**  1.087(0.015)**
controls 3  0.666(0.098)**  0.838(0.11)
controls 4  0.958(0.122)    1.129(0.127)
controls 5  0.706(0.144)**  0.750(0.146)**
controls 6  1.012(0.086)    0.998(0.089)
controls 7  0.957(0.141)    1.010(0.142)
controls 8  0.694(0.173)**  0.726(0.18)*
controls 9  1.895(0.065)**  1.575(0.072)**
controls 10 1.353(0.085)**  1.279(0.095)**
controls 11 0.907(0.083)    0.949(0.095)
controls 12 0.988(0.135)    0.920(0.145)
Log Likelihood  -3417.879   -3356.028
AIC                   6863.758  6754.056
BIC                   6958.156  6895.653
PCP                     0.702   0.697

Answer 1

I am a little off topic because I just offer you an other example of this kind of behavior, but hopefully this is extreme enough to help you understanding it.

So my point is that when a factor associated with a trait is frequent, it can be a terrible predictor, even if the association is strong. Here is R code for an hypothetic case/control study. First, generate y and x:

runif(1)
y <- as.factor( c( rep(0,100), rep(1,100) ) )
x <- as.factor( c( sample(0:1, 100, replace=TRUE),
                   sample(0:1, 100, replace=TRUE, prob=c(1,99) )))

Then run a logistic regression:

> summary(glm(y ~ x, family=binomial(link=logit)))

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-1.4916  -0.2747   0.3091   0.8929   2.5674

Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept)  -3.2581     0.7203  -4.523 6.08e-06 ***
x1            3.9719     0.7415   5.357 8.48e-08 ***

The $\beta_1$ value is huge, isn’t it? And the $p$-value is tiny. Don’t pay attention to the intercept, this is a case/control study.

As huge as $\beta_1$ is, $x_1$ is a terrible predictor: of course, 99% out of the cases ($y=1$) have $x=1$, but still 50% of controls have $x=1$. So if you use $x=1$ to predict $y=1$, you will misclassify 50% of the controls (and if the trait $y=1$ is rare, near to 50% of the population).

You can modify this example to get the reverse behavior (correctly classifying controls but missing many cases), etc. You will even get a terrible behavior is all categories like this :

runif(1)
y <- as.factor( c( rep(0,100), rep(1,100) ) ) 
x <- as.factor( c( sample(0:1, 100, replace=TRUE, prob=c(7,3)),
                   sample(0:1, 100, replace=TRUE, prob=c(3,7) )))

Let's go:

> summary(glm(y ~ x, family=binomial(link=logit)))

Deviance Residuals: 
      Min        1Q    Median        3Q       Max  
-1.54527  -0.85630  -0.00329   0.84972   1.53697  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -0.8145     0.2157  -3.775  0.00016 ***
x1            1.6474     0.3072   5.363  8.2e-08 ***

The $\beta$ is still quite large, but you can’t classify with this variable...

PS See also this paper (in Genetics)

Answer 2

You're not alone; lots of us feel embarrassed trying to explain to our audiences that, even with stat. sig. results, we still can't classify cases correctly very often. But remember that classification is partially under your control. You don't have to accept your software's probable default of 0.5 as a threshold for the predicted probability that will get a case classified as a predicted "1." Maybe in your context you want to set that threshold (cut point) much lower. Studying an ROC curve will help you decide what your options are and how you want to handle the tradeoff between sensitivity (identifying a large enough fraction of the true positives) and lack of specificity (ending up with too many false positives).

Answer 3

Statistical significance is not all that significant, in many cases. Effect sizes are more meaningful.

One way the results you describe can happen is if there is a large sample. This makes even small effects significant.

Since you are looking at logistic regression models, I would examine the odds ratios and see if they are interestingly large.