I'm new to R and logistic regression and have to admit that I don't really know how to interpret the result. I'm trying to compute a pretty simple model with 2 predictors (A and B). When I first try to compute models with the predictors one by one they are both significant. When I put them together and add an interaction term they lose their significance (but the interaction term is weakly significant). I interpret this as A and B are overlapping and no longer significant when the oter parameter is hold constant. Right?

But now to the part I don't know how to interpret. I make predictions from my models (see code below) and then run t-tests for the predictions vs. the depending variable. I think this should give a hint on how good the model is (is there a better way?). When I do it this way I get a much lower p-value for the model with both A and B. I think this is contradictory. The first part tells me that A doesn't provide any significant information to the model when combined with B, but on the other hand I get much better predictions. I guess something is really wrong, but I can't figure out what. Can you help me?

model1=glm(f~A, , family=binomial(link="logit"))
model2=glm(f~B,   family=binomial(link="logit"))
model3=glm(f~A*B, family=binomial(link="logit"))
p1=predict(model1, newdata=data, type="response", na.rm=TRUE)
p2=predict(model2, newdata=data, type="response", na.rm=TRUE)
p3=predict(model3, newdata=data, type="response", na.rm=TRUE)

Part of the output:

> summary(model1)
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -1.9756     0.3499  -5.647 1.64e-08 ***
A            -0.5898     0.2119  -2.784  0.00537 ** 

> summary(model2)
              Estimate Std. Error z value Pr(>|z|)  
(Intercept)  8.354e-01  1.309e+00   0.638   0.5234  
B           -1.028e-04  5.122e-05  -2.007   0.0447 *

> summary(model3)
              Estimate Std. Error z value Pr(>|z|)  
(Intercept)  1.254e+00  1.705e+00   0.735    0.462  
A            1.589e+00  9.743e-01   1.631    0.103  
B           -1.324e-04  7.333e-05  -1.805    0.071 .
A:B         -9.418e-05  4.632e-05  -2.033    0.042 *

> t.test(p1~f)
t = -2.614, df = 11.83, p-value = 0.02286

> t.test(p2~f)
t = -1.8702, df = 15.679, p-value = 0.08024

> t.test(p3~f)
t = -4.9777, df = 17.344, p-value = 0.0001084
  • 1
    $\begingroup$ Are you certain that you want to use A*B and not A+B? Using the t-test is interesting. Have you also calculated precision and recall? $\endgroup$ – John Yetter Mar 21 '14 at 17:37
  • $\begingroup$ Note that model1 has a typo. There is either an extra comma , or a missing argument (but then the argument is presumably missing in the other models as well). $\endgroup$ – gung - Reinstate Monica Mar 21 '14 at 17:40

There are several issues here:

  1. As @John Yetter notes, you have included the interaction A*B in model3 as well as the two variables A & B. If you only wanted to fit a model with the two variables, you would use: glm(f~A+B, family=binomial(link="logit")).

    If you do want the interaction / when you have one included in the model, the interpretation of the main effects differs. Specifically, the main effect of A is the relationship between A and f when B is equal to 0 (and vice versa for B).

  2. The relationship between regressing y on x, and regressing x on y is, in general, not the same. This is especially true for logistic regression (where there is a non-linear transformation in between y and x, but not between x and y) and for multiple regression (logistic or otherwise). That is, you cannot turn around your first model f~A into a t-test A~f.

  3. You should not test the relationship between the model's predicted values and the original response value as a way to determine "how good the model is". This makes no sense. Since the predicted values are a perfect (i.e., non-stochastic) function of the predictor variables, there is no new / different information there. A more typical way to understand the quality of the model is to examine the ROC curve.

  • $\begingroup$ Thanks for answering! 1) I tried with and without the interaction term and the reason for choosing to include it is a better outcome (sensitivity/specificity). $\endgroup$ – user42346 Mar 31 '14 at 19:30
  • $\begingroup$ Thanks for answering! 2) I don't understand this part. When I try to turn the t-test around (f~A) I get the error message "grouping factor must have exactly 2 levels". I agree that it looks weird when they aren't in the same order, but that's the only way to avoid error messages. How should it be? $\endgroup$ – user42346 Mar 31 '14 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.