2
$\begingroup$

I am running a logistic regression to study the association between a three-level factor and a binary outcome, after controlling for covariates. My sample is small to medium (~800). The likelihood ratio test of adding the factor is not significant which, to my understanding, indicates the model without the factor fits the data as well as the model with the added factor, and therefore the factor can be dropped. Yet, the odds ratio of at least one level of the factor is significant (the profile CI does not include 1). I have seen people do the following and am wondering which is on better statistical footing:

  1. Report both the nonsignificant LRT and interpret the odds ratios of the factor levels. I have seen people argue that LRT may not be significant due to power and sample size and/or argue that the odds ratio should still be interpreted as they test differences between levels even if the overall factor is not significant. Some folks also interpret the Odds Ratios but then hedge (report everything but then throw in a line that it could be an artifact).
  2. Report only the nonsignificant LRT and do not interpret the significant odds ratios for the factor levels, under the rationale that they do not explain the data better and significant findings regarding levels of that factor could be artifacts.

I am leaning toward the second, yet would appreciate advice and sources which discuss whether to interpret significant findings for factor levels when the LRT is not significant. I see this as similar to the debate on this post whether to interpret comparisons or contrasts if an omnibus test is not significant. I see other posts discussing the discrepancy, just not the best way to interpret it.

EDIT: For others in the same situation (i.e., the LRT between logistic models after adding a factor is not significant, yet the odds ratio for a level of the factor doesn't include 1 and therefore looks significant)), based on the advice below and after reading other papers, I ran the factor level comparisons using Tukey post-hoc test with Holm-adjusted p values to adjust for running multiple comparisons (instead of just looking at the profile CIs). In all the cases were the LRT was not significant, the adjusted p values for factor levels were also not significant. So, this method (LRT between models along with post-hoc tests with adjusted p values) appears more consistent and makes it easier to conclude that any difference, implied by the odds ratio CI, between the factor level and the reference group is likely either small or an artefact. For others wondering how to conduct these post-hoc test for a logistic regression, you can use:

library(multicomp)
summary(glht(model, mcp(factor_of_interest="Tukey")), test=adjusted(type="holm"))


See here for more info on how to conduct and write this up.

$\endgroup$

1 Answer 1

1
$\begingroup$

Basically, this comes down to a tradeoff between false positive and false negative errors, both for you and your audience. So long as you are straightforward in explaining just what you did, either approach in principle is justifiable. The traditional "p < 0.05" cutoff for "significance" is arbitrary to begin with; an honest interpretation of what the data and the model might mean when generalized to a population is what's called for. Use your understanding of the subject matter and the expectations of your audience to guide how you proceed.

A potential "gotcha" here is that, in standard coding of a 3-level categorical predictor, "the odds ratio of at least one level of the factor is significant" in terms of a difference from your choice of the reference level. The apparent "significance" of a coefficient in this case can depend on the reference level. So proceed cautiously.

Finally, note that the comment from Russ Lenth on this answer spoke of "Multiple comparisons or contrasts with responsible adjustments for multiple testing at the simultaneous-CI protection (e.g., Tukey)" when he argued against the need for passing an omnibus test first. So if you are going to try to claim "significance" for a particular coefficient, make sure that you are taking appropriate precautions in that regard. That's particularly important if you are now examining that particular coefficient just because it showed up as "significant."

$\endgroup$
1
  • $\begingroup$ Thank you. Great point about making sure to account for multiple comparisons $\endgroup$
    – Mark
    Commented Apr 19, 2021 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.