1
$\begingroup$

I'm running some regression models, but I'm finding that my pattern of results change depending on the reference groups. My predictors are Condition(A vs. B), Status (low vs. high), and Age (continuous, mean-centered).

In R, my first model is: score ~ condition x status x age

Results when reference groups are condition = A, status = low: Ref A-low

Results when reference groups are condition = A, status = high: Ref A-high

Results when reference groups are condition = B, status = low: Ref B-low

Results when reference groups are condition = B, status = high: Ref B-high

Plot of the raw data:

Raw data figure

I understand that changing the reference groups will change the coefficients, but I assumed the t-values and p-values would remain the same since I have dichotomous variables. There isn't a strong theoretical basis for using one reference group over the other. Looking at a plot of the raw data, it looks like there should at least be main effects of condition, status, and age, as well as an interaction of condition*status. Why am I getting different results and how might I address this issue?

I'm also running these same models on different DVs and sometimes the results are the same regardless of the reference groups and sometimes they differ. I have this same issue in other mixed-effects models where I just have an addition dichotomous within-group predictor so I have the participant ID included as a random effect.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I understand that changing the reference groups will change the coefficients, but I assumed the t-values and p-values would remain the same since I have dichotomous variables.

The coefficient t-tests are for determining whether the estimated value of a coefficient is significantly different from 0. That's not a test of the overall significance of a predictor.

When a predictor is involved in interactions, its individual coefficient values and those for its lower-level interactions are estimated for a situation when all its interacting categorical predictors are at their own reference levels.* So changing the reference level of a categorical predictor can change the value of coefficients involving other predictors with which it interacts, and thus the "significance" of their differences from 0.

Despite the apparent differences in coefficient "significance," any predictions made from those models will be identical. The models are fundamentally the same. Only the ways that initial summaries are shown are different. If you want to evaluate the overall significance of any predictor, you can do likelihood-ratio tests of nested models with and without it, or perform Wald tests that evaluate together all coefficients involving it.

*This explanation is for the default R treatment/dummy coding of predictors.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you @EdM. I used LRTs to determine the best model fit for each DV (including with the mixed effects models). My results now seem more consistent with the raw data and regardless of whether I change the reference level. Is it always ideal to determine the best model fit for each DV first? In my field, it's common to use the full models unless a predictor is not significant in any model. Now that I'm using more parsimonious models, I no longer have significant 3-way interactions. Even though they came out as significant before, were they essentially meaningless? $\endgroup$
    – kmy221
    Commented Feb 19, 2022 at 19:55
  • $\begingroup$ @kmy221 with multiple DV it's usually best to do a true "multivariate regression" (multiple-outcome) that takes correlations among outcomes into account, with the same predictors for all outcomes. This page is one example. If your interest is in prediction, it's generally best to keep all predictors unless you are overfitting the data. Chapter 4 of Harrell's course notes and text discuss the parsimony/complexity tradeoff. $\endgroup$
    – EdM
    Commented Feb 20, 2022 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.