2
$\begingroup$

First, I'm really sorry if this is a duplicate, I just couldn't find a definite answer.

I'm not used to analyzing categorical variables so I'm having trouble with something that is probably very basic. So, I'm running a glmer model with a binary outcome and 2 categorical predictors, one with 2 levels (pub) and one with 4 levels (tcat):

model<-glmer(outcome ~ (1|loc)+pub*tcat, data=data1, family="binomial")
summary(model)
<snip>
Fixed effects:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)  -0.8210     0.5142  -1.597   0.1103  
pub          -1.1550     0.4707  -2.454   0.0141 *
tcat2        -0.2438     0.5195  -0.469   0.6389  
tcat3         0.2555     0.5319   0.480   0.6309  
tcat4         0.2347     0.6147   0.382   0.7026  
pub:tcat2     0.7972     0.4856   1.642   0.1007  
pub:tcat3     0.6310     0.4917   1.283   0.1994  
pub:tcat4     0.8240     0.5478   1.504   0.1325

So, none of the interactions are significant here. However, when I look at the contrasts:

em1<-emmeans(model, pairwise ~ pub|tcat)
em1$contrasts
$contrasts
tcat = 1:
 contrast estimate    SE  df z.ratio p.value
 1 - 2       1.155 0.471 Inf   2.454  0.0141

tcat = 2:
 contrast estimate    SE  df z.ratio p.value
 1 - 2       0.358 0.127 Inf   2.827  0.0047

tcat = 3:
 contrast estimate    SE  df z.ratio p.value
 1 - 2       0.524 0.145 Inf   3.619  0.0003

tcat = 4:
 contrast estimate    SE  df z.ratio p.value
 1 - 2       0.331 0.281 Inf   1.178  0.2389

The contrasts suggest that "pub" levels 1 and 2 differ within tcats 1,2, and 3, but not within tcat=4. Right? However, should I even look at the contrasts when the interaction in the fixed effects part is not significant? I got the impression from some helpful answers I found that the information provided by the contrasts kind of "overrides" the information provided the fixed effects estimates but I couldn't confirm that. Working with continuous variables I learned (perhaps incorrectly?) that if a cont x cont or a cat x cont interaction is not significant, you don't go testing the simple slopes, you accept it and move on but I'm confused now because I have gotten the impression it doesn't work this way with factorial predictors.

I also ran a model comparison ANOVA between the above model and one without the interaction, and it was non-significant (Chisq. = 3.6, p = .308).

Any insight would be appreciated!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You don't need to have a "significant" overall result to do specific comparisons, if those comparisons were planned initially. See this page. That's the case for continuous or categorical predictors, despite what you might have inferred from prior study. As far as the modeling is concerned, categorical predictors are numerical predictors, just with a very restricted range of values (0/1). A global test has an advantage in discouraging you from excessive data snooping, which is probably why you have been cautioned against further analysis when the interaction isn't "significant." But if the tests were pre-planned that's not an issue.

It comes down to what hypothesis you are trying to test and how you will interpret your results. If your initial hypothesis was whether the pub contrasts were significantly different from 0 within each of the tcat levels, then what you did is fine.

Don't over-interpret those results, however. The apparent lack of "significance" of the interaction term or of the pub contrast within tcat=4 might just be due to an under-powered study.

Compare the pub contrasts within tcat=2 and tcat=4. The point estimates are very close (0.358 vs. 0.331), but the standard errors are quite different (0.127 vs. 0.281). So by the usual p < 0.05 criterion of statistical significance, the pub contrast within tcat=4 isn't "significant" even though its estimated value isn't far from that of tcat=2. On that basis, do you really want to conclude that there is no effect of pub within tcat=4?

Within tcat=1, the point estimate of 1.155 for the pub contrast is numerically much higher than those within the other levels, but its standard error of 0.471 is much larger. The standard errors of coefficients are in part determined by the numbers of observations: were there many fewer observations having tcat=1? Do you then want to interpret the lack of "significance" of the interaction term (as best evaluated by anova() on the models with and without the interaction; don't rely on the individual interaction coefficients) as a true lack of differences among the tcat levels, or just as a sign that your study wasn't large enough?

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you SO MUCH for taking the time to answer, it was tremendously helpful and exactly what I needed to know! And thank you for linking the older post, I hadn't found it though I looked. And you are of course correct about looking at the estimate sizes and standard errors more closely, I just got stuck with the not understanding how to proceed with the interaction vs. contrasts. I'll definitely go through them in detail. $\endgroup$
    – Sointu
    Jan 20 at 22:20
  • $\begingroup$ @Sointu I often find it helpful to sort search results by "score" on this site rather than by the default "relevance." Some of the top-scored hits might not be relevant to your search, but going down the list by "score" is more likely to give you a well respected and comprehensive answer to your query. $\endgroup$
    – EdM
    Jan 20 at 22:24
  • $\begingroup$ Thanks for the tip - I think I got kind of overwhelmed with the amount of emmeans resources, though I'm grateful they exist. $\endgroup$
    – Sointu
    Jan 20 at 22:25
  • $\begingroup$ If you don't mind - a small follow-up question: what if the study is exploratory and we don't have strong hypotheses, and there are, say, more predictors, all of which interactions we would like to check (of course we would not make strong claims in this case, but we'd like to say something like "this was more likely under these conditions in the current dataset"). In your opinion would it be OK to use contrast differences as basis of our reasoning in this case (of course with p-value adjustments), or would it be better to just forget all testing and stick with e.g. visualizations of data? $\endgroup$
    – Sointu
    Jan 20 at 22:39
  • 1
    $\begingroup$ @Sointu graphical visualizations and contrast differences are both fine for exploration, provided that you don't try to make claims based on "significance" of the contrasts. I'd be reluctant even to say things like "more likely"; let the results speak for themselves and emphasize the implications of your results (point estimates and standard errors) for design of subsequent more-definitive studies. 6800 sounds like a lot of observations for an "exploratory" study; what matters most in binomial regression is usually the number of cases in the minority class. $\endgroup$
    – EdM
    Jan 21 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.