Handling baseline differences with a retrospective study and mixed model

Question

I am looking at the fixed effects of Var1, Var2, and Var3 on a dependent variable V with a random effect across participant ID using the lmer function in R.

• Var1 has levels TimePoint1, TimePoint2, and TimePoint3
• Var2 has levels TreatmentA and TreatmentB
• Var3 has levels LocationX and LocationY

My mixed model has the following form:

V ~ Var1 * Var2 * Var3 + (1 | ID)

which showed a main effect of Var1, Var2, and Var3 with no significant interactions. I thus removed the interactions and repeated the model in the following way:

V ~ Var1 + Var2 + Var3 + (1 | ID)

which showed the same main effects.

However, here's the catch. Factor Var2 refers to treatment type, of which the levels are TreatmentA and TreatmentB. It turns out that upon review of baseline characteristics, I found that the ages are significantly different by t-test between people in group Var2=TreatmentA and people in group Var2=TreatmentB, but only when Var1=LocationX (and not when Var2=LocationY)

Thus, I am thinking I need to include a covariate Age in my original mixed model. However, since Age is only significantly different across the levels of Var2 when Var1=LocationX, how should I structure my mixed model to handle this? I was thinking to just add an interaction term between Age and Var2.

V ~ Var1 + Var2 + Var3 + Var1:Age + (1 | ID)

Thoughts? I could also extract out the subset of my data when Var1=LocationX and then make a mixed model just for that data with a covariate of Age. I'm not really sure what the appropriate approach is.

Answer 1

Incorporating covariates like age into your model should be based primarily on how they are expected to be related to outcome. For example, if knowledge of the subject matter suggests that age is related to outcome on its own but that its relation to outcome doesn't depend on the other variables, then it should be incorporated on its own without any interactions. If, however, you expect for example that the effect of age on outcome will differ with treatment, then you should include an age-treatment interaction. Unless you think that the relation of age to outcome depends on Location then you shouldn't be including an interaction of age:Location.

If you are primarily interested in the effect of Treatment and you are still concerned about lack of balance in baseline covariate differences, you could consider analysis that incorporates propensity scores. The propensity score relates the probability of treatment-group membership to the combination of covariate values, for example by logistic regression or by tree-based approaches as provided by the R package twang. Provided that there is overlap in the baseline characteristics for the two treatment groups, then you can get balance by selecting cases that received the 2 treatments matched on their propensities, or you can use the inverses of the propensity scores as weights in your regression. Regressions with mixed models can incorporate weights.

Inverse propensity score weighting can be done along with incorporating age and other covariates into the model proper, to give a doubly robust model.

With respect to specifics of your data set, you say

ages are significantly different by t-test between people in group Var2=TreatmentA and people in group Var2=TreatmentB, but only when Var1=LocationX (and not when Var2=LocationY)

and in your last model you propose to adjust for this by including an age:Location interaction without a main term for age. There are a couple of issues with this.

First, including an interaction without a main effect for each of the variables is generally not good practice and at best makes interpretation of the coefficients troublesome; see this page for extensive discussion. You could, however, add a main effect of age along with this interaction, and that would be OK if justified by your understanding of the subject matter.

Second, it's not clear that this age:Location interaction really controls for an important issue at hand. If a linear model is valid (e.g., outcome linearly related to age), a model with that interaction and with age itself would correct for overall differences in outcome due to age, and differences between locations with respect to the influence of age, but it wouldn't correct for any influence of treatment on outcome that depends on age. I suspect that an age:treatment interaction would be even more of an issue than the age:Location interaction, particularly if clinicians at one location seem to be preferring one treatment over another based on age. So think carefully about what interactions make sense to include based on your understanding of factors that are contributing to outcome overall, and don't be focused in your model solely on the particular age mismatch between treatments at one location. Again, propensity score methods can be useful in dealing with this type of covariate imbalance to provide a model that can predicts how outcomes would differ between the two treatments if applied to the entire population.

Subsetting typically loses information by leaving informative cases out of the analysis. If a linear model including clinically relevant interactions (e.g., with Location or Treatment as needed) is well specified and calibrated then a combined model will take advantage of the greater number of cases to provide more precise estimates of the coefficients than will subsetting.

Answer 2

A couple of points:

• I would advice against looking at the results of the t-tests to determine which variables and/or interactions to put in your model. These will suffer from multiple testing issues and may give you wrong impression on which variables seem to be related to your outcome. Moreover, to stably identify interaction terms with sufficient precision, you typically need large sample sizes. You have not mentioned how many subjects you have.
• In mixed models a general model-building strategy is a follows:
  • You start with a “full” specification of your fixed effects. There you need to correct for variables that you think are associated with your outcome. There you should consider potential nonlinear terms for your continuous covariates, and interaction terms, taking into account the last sentence in the previous bullet point.
  • Then you build you random-effects structure. This is important because correct/efficient inference for the fixed effects depends on a correct/appropriate specification of the random-effects structure. In longitudinal studies, as it seems is the one you have, you start with random intercepts, and then you include random slopes, testing if they improve the fit, and moving to high order nonlinear terms if required.
  • When you have selected you random effects, you return to the fixed effects and see which variables seem to be associated with your outcome. You can start with omnibus tests to see if you can drop the interactions terms all together (using a bit higher significance level, e.g., 0.1 or 0.15), and doing likewise for the nonlinear terms. If these terms seem not to substantially improve the fit, you could drop them, and keep the full additive model in which you should not drop any other covariates.
• If you are interested, you can have a look about these things in the note of my Repeated Measurements course.

Answer 3

Building a model by considering significance of predictors/their interactions is a generally poor policy that leads to poor properties of estimators and confidence intervals, a lack of generalizability/out of sample performance and so on. Whether something is "significant" or not does not really result in any meaningful properties of the predictor: it can still be incredibly important despite being "non-significant", "significance" can arise in a very spurious fashion and can be practically irrelevant. It matters much more whether there are considerations (theoretical/biological/based on how treatment decisions are made/etc.) why predictors/interactions should matter, whether the influence of a variable is so striking that you cannot ignore it and/or whether cross-validation shows improved performance (in terms of whatever you care about - the third criterion should hopefully cover the second one).

Another important question is whether this is confirmatory work (i.e. you want to publish the results of this and claim something e.g. with a p-value) or not. If it is, then you should really have a single pre-specified model, in which you perhaps specify a bunch of things you are not sure are needed, but which you leave in the model just in case they are needed. On the other hand, if you are trying to generate hypotheses/models/algorithms/whatever and then want to test them in a new experiment (or on a hold-out test set that you have made very sure to not look at while building your model) after you have decided on what you consider your most promising model, then you have a lot more freedom (but probably still want to do something like cross-validation/assessment on a validation set to not fool yourself).

Additionally, for comparing treatments something like propensity scores (as mentioned by @EdM) or structural equation models (if treatment decisions happen at multiple occasions) are well-known alternatives to regression adjustments.

I actually think it would be no problem to have a propensity score (or regression adjustment) that includes a location by age interaction, because if the treatment assignment mechanism simply includes these E.g. doctors in location a are more likely to assign a certain treatment to the elderly than doctors in the rest of the country, or if the middle aged people have worse outcomes in one part of the country than another, then those things are simply the sort of things that might be the case. What we care about is the treatment effect on top of that, so we try to adjust for these things. It becomes a different matter when we talk about treatment by location/age interactions. Those really lead to a different interpretation (if the interactions are not really small) that would imply that the treatment works different well by locations/age. That may be true and may be worth investigating, but it is almost a different question than the question for the average treatment effect adjusting for other factors.