# Significant model with no significant betas?

I am running 3 models on 3 subsets of the same data. The set up is as follows:

1. Outcome (DV) is binary categorical
2. Time (IV) is repeated twice (pre and post)
3. Treatement (IV of interest) is binary categorical

I am interested to know if at time 2 treatment has had an effect on outcome. I used the lme4 package and used the following R code:

tot.null<-lmer(as.factor(outcome)~Time+(1|id), family=binomial(link='logit'),
data=df.total)
tot.mod<-lmer(as.factor(outcome)~trt*Time+(Time|id),
anova(tot.null,tot.mod)
summary(tot.mod)


   id             trt Time outcome
1   1 peer discussion   -1       1
2   2 peer discussion   -1       1
3   3 peer discussion   -1       0
4   4 peer discussion   -1       1
5   5 peer discussion   -1       1


str of data

> str(df.total)
'data.frame':   872 obs. of  4 variables:
$id : int 1 2 3 4 5 6 7 8 9 10 ...$ trt    : Factor w/ 2 levels "peer discussion",..: 1 1 1 1 1 1 1 1 1 1 ...
$Time : num -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 ...$ outcome: num  1 1 1 1 1 1 1 0 1 0 ...


The problem is I get an error messoge on the tot.mod:

> tot.mod<-glmer(as.factor(outcome)~trt*Time+(Time|id), family=binomial(link='logit'),
data=df.total)
Warning message:
In mer_finalize(ans) : false convergence (8)


I think this is the reason the model is significant but none of the predictors are. look at the inflated SEs.

Comparison to the null model and the summary of full model

> anova(tot.null,tot.mod)
Data: df.total
Models:
tot.null: as.factor(outcome) ~ Time + (1 | id)
tot.mod: as.factor(outcome) ~ trt * Time + (Time | id)
Df    AIC    BIC  logLik  Chisq Chi Df            Pr(>Chisq)
tot.null  3 689.54 703.85 -341.77
tot.mod   7 410.67 444.07 -198.34 286.86      4 < 0.00000000000000022 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> summary(tot.mod)
Generalized linear mixed model fit by the Laplace approximation
Formula: as.factor(outcome) ~ trt2 * Time + (Time | id)
Data: df.total
AIC   BIC logLik deviance
410.7 444.1 -198.3    396.7
Random effects:
Groups Name        Variance Std.Dev. Corr
id     (Intercept)  396.46  19.911
Time        1441.98  37.973   0.470
Number of obs: 872, groups: id, 436

Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 10.09866    3.33921   3.024  0.00249 **
trt21        0.01792    5.10796   0.004  0.99720
Time        -0.93753    5.79560  -0.162  0.87149
trt21:Time  -0.84882   10.41073  -0.082  0.93502
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
(Intr) trt21  Time
trt21      -0.654
Time        0.558 -0.365
trt21:Time -0.311  0.473 -0.557


What's going on? Why is the model significant but none of the betas? In OLS I know this is an indicator of multi-colinearity among predictors. I don't think that's the reason here. Please help with understanding this problem as well as the error message (I think they may be connected). What are some things I should check for?

The other two models from the same data set (split on a different grouping variable) had no apparent problems.

Using R 2.14.2, lme4 v. 0.999375-42 on a win 7 machine

-

There are (at least) two issues here.

The first is that the models you're comparing differ by two predictors and a random effect, not just a single predictor. You need to test one thing at a time. So, there may be no effect of trt, or an interaction, but the two may be contributing enough that there is a significant effect with neither of them explaining a substantial portion of the variance by itself.

The second is that when you put an interaction in the model then you can no longer see the pure main effects. Try doing:

tot.add <- lmer(as.factor(outcome) ~ trt + Time + (Time|id),
family = binomial(link='logit'), data = df.total)


You might, but I'm in no way guaranteed this, see an effect of trt. You also need to check with and without Time as a random effect and really have a better handle on what you're trying to test.

The results of these might give you a better handle on what is going on...

(and for goodness sakes ALWAYS put spaces around the assignment operator... what does a<-2 mean?.. a <- 2 or a < -2?)

EDIT: OK, sounds like you want an ANCOVA like test?

Just look at that model I put in above. The beta weight will be a good indicator of whether there's a significant effect of Time when trt is accounted for in outcome for the additive model. (this assumes no time x trt interaction)

-
Can you explain some possibilities of why the SEes would be so inflated. –  Tyler Rinker Mar 18 '12 at 17:53
Inflated compared to what? I see one set of SE's in your question. Did you look at the model I mentioned? What exactly are you trying to test anyway? Your anova() above lacks specificity. –  John Mar 18 '12 at 20:40
Yes I did examine that model. It is significant over the null model. My test was that at time two the treatment group would have more yes outcomes than the comparison group while controlling for pretest scores. I don't quiet understand the comment of your anova() lack specificity. –  Tyler Rinker Mar 19 '12 at 0:21
I stated in my answer... you compared a model with time only to one with trt, time:trt, and time as a random effect. You need to make models that test each of these one at a time to have any idea what causes what. I gave you an example of a model you can compare to tot.mod. Furthermore, the betas of main effects of models with an interaction don't really mean anything with respect to the question, "does this predictor make a significant contribution?" –  John Mar 19 '12 at 1:07
I'm not sure what you mean by "controlling for pretest scores". If you're attempting an ANCOVA then specify that. In that case you don't care about an interaction at all and what you want to see is what's in my answer. –  John Mar 19 '12 at 1:22
show 1 more comment