0
$\begingroup$

Is it meaningful to interpret the contrast result from a post-hoc test on factor levels that have non-significant coefficients in the fitted mixed model regression?

I'm using a contrast test using emmeans package to see if there is any significant difference between the treatment groups in my analysis. But the contrast test on a couple of factor levels does not make sense (I don't go to the details)! These are the factor levels that had non-significant coefficients in my GLMM model. I couldn't find any reason for it. So, my question is whether interpreting post-hoc test is meaningful when the coefficients are not significant. And should I ignore those factor levels??

Edits:

Here is the fitted model:

library(rstanarm)
library(emmeans)

fit1 <- stan_glmer(Response ~ 1 + Treatment + Time + (1 | Replicate),
                   family = binomial(link = "logit"),
                   cores = getOption("mc.cores", 4L),
                   data = mydata)

# calculating the mean differences
emm2 <- emmeans(fit1, specs = pairwise ~ Treatment | Time)

These are the coefficient estimates and the contrast results. The part that does not make sense is the difference between the proportion of the response in the Treat1 and Treat3 is much higher than those from the Treat2. But the Treat2 has a significant contrast with both Treat1 and Treat3, while the Treat1-Treat3 contrast is non-significant. If you notice, both Treat1 and Treat2 have also non-significant coefficients in the original mixed model.

enter image description here

$\endgroup$
3
  • 1
    $\begingroup$ I suggest adding more information. Perhaps including the model and the contrast statement you are testing. $\endgroup$
    – R Carnell
    Feb 8, 2021 at 1:30
  • 1
    $\begingroup$ Show us some output and tell us what you think is wrong. $\endgroup$
    – Russ Lenth
    Feb 8, 2021 at 1:35
  • 2
    $\begingroup$ Generally, if the overall null hypothesis (no differences) is not rejected one should have good prior reasons for looking for specific differences. Even with traditional protections against false discovery, superfluous ad hoc comparisons should be avoided. They often use somewhat different distribution theory than the main test and so one may inadvertently underestimate the risk of false discovery. $\endgroup$
    – BruceET
    Feb 8, 2021 at 7:27

2 Answers 2

0
$\begingroup$

It is certainly meaningful. The specific contrasts have exactly the same meaning, regardless of whether the overall test is significant or not, or even whether you do a main test at all.

Of course, doing multiple tests increases your type 1 error. But, then, it decreases your type 2 error (most things that increase one decrease the other). In some cases, type 2 errors are much worse than type 1.

And you have to be honest about what you did when you report it.

$\endgroup$
-1
$\begingroup$

For your specific question : it is not meaningful in general to interpret posthoc result if the omnibus test is not significant. You are increasing your type I error rate, misspecifying your data, as well as violating your main hypothesis. However, it can be meaningful if the posthoc analysis answers a specific a priori hypothesis. It has to be carried adequately though.

$\endgroup$
2
  • 4
    $\begingroup$ I disagree. There's no law that says an omnibus test is needed at all. Omnibus tests are indeed useful, but primarily for model selection. Multiple comparisons or contrasts with responsible adjustments for multiple testing at the simultaneous-CI protection (e.g., Tukey) do not require omnibus tests. Indeed, the standard F test is equivalent to testing all possible contrasts using the Scheffe procedure. So testing only a finite number of contrasts with the Scheffe method is conservative, even if the F test is not conducted. $\endgroup$
    – Russ Lenth
    Feb 8, 2021 at 4:14
  • 1
    $\begingroup$ I 100% agree with @Russ Lenth, that is why i specified in general and in pecific contexts. $\endgroup$
    – POC
    Feb 8, 2021 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.