I am looking at the fixed effects of Var1, Var2, and Var3 on a dependent variable V with a random effect across participant ID using the lmer function in R.

  • Var1 has levels TimePoint1, TimePoint2, and TimePoint3
  • Var2 has levels TreatmentA and TreatmentB
  • Var3 has levels LocationX and LocationY

My mixed model has the following form:

V ~ Var1 * Var2 * Var3 + (1 | ID)

which showed a main effect of Var1, Var2, and Var3 with no significant interactions. I thus removed the interactions and repeated the model in the following way:

V ~ Var1 + Var2 + Var3 + (1 | ID)

which showed the same main effects.

However, here's the catch. Factor Var2 refers to treatment type, of which the levels are TreatmentA and TreatmentB. It turns out that upon review of baseline characteristics, I found that the ages are significantly different by t-test between people in group Var2=TreatmentA and people in group Var2=TreatmentB, but only when Var1=LocationX (and not when Var2=LocationY)

Thus, I am thinking I need to include a covariate Age in my original mixed model. However, since Age is only significantly different across the levels of Var2 when Var1=LocationX, how should I structure my mixed model to handle this? I was thinking to just add an interaction term between Age and Var2.

V ~ Var1 + Var2 + Var3 + Var1:Age + (1 | ID)

Thoughts? I could also extract out the subset of my data when Var1=LocationX and then make a mixed model just for that data with a covariate of Age. I'm not really sure what the appropriate approach is.

  • $\begingroup$ I'm not going to attempt an answer as your question is just about at the limit of my knowledge. But I do have a question about why you're thinking to introduce age as a potential interaction with Var2, rather than simply having age as a no-interaction covariate. Usually we think of an interaction in cases where the effect of age on your outcome (V) differs for levels of Var1. $\endgroup$ Jan 21 '19 at 22:27
  • 1
    $\begingroup$ How were the treatments determined? Were they randomised? (As a general rule, post-hoc "balance" tests on randomised treatments are a bad idea.) $\endgroup$
    – Ben
    Jan 22 '19 at 1:05
  • $\begingroup$ The treatments were not randomized but were determined clinically in a complex way. $\endgroup$
    – CodeGuy
    Jan 22 '19 at 14:16
  • $\begingroup$ @BrentHutto I guess I was trying to be conservative. Suppose that one of the original interactions that I dropped from the original model actually did depend on Age. $\endgroup$
    – CodeGuy
    Jan 22 '19 at 15:32

Incorporating covariates like age into your model should be based primarily on how they are expected to be related to outcome. For example, if knowledge of the subject matter suggests that age is related to outcome on its own but that its relation to outcome doesn't depend on the other variables, then it should be incorporated on its own without any interactions. If, however, you expect for example that the effect of age on outcome will differ with treatment, then you should include an age-treatment interaction. Unless you think that the relation of age to outcome depends on Location then you shouldn't be including an interaction of age:Location.

If you are primarily interested in the effect of Treatment and you are still concerned about lack of balance in baseline covariate differences, you could consider analysis that incorporates propensity scores. The propensity score relates the probability of treatment-group membership to the combination of covariate values, for example by logistic regression or by tree-based approaches as provided by the R package twang. Provided that there is overlap in the baseline characteristics for the two treatment groups, then you can get balance by selecting cases that received the 2 treatments matched on their propensities, or you can use the inverses of the propensity scores as weights in your regression. Regressions with mixed models can incorporate weights.

Inverse propensity score weighting can be done along with incorporating age and other covariates into the model proper, to give a doubly robust model.

With respect to specifics of your data set, you say

ages are significantly different by t-test between people in group Var2=TreatmentA and people in group Var2=TreatmentB, but only when Var1=LocationX (and not when Var2=LocationY)

and in your last model you propose to adjust for this by including an age:Location interaction without a main term for age. There are a couple of issues with this.

First, including an interaction without a main effect for each of the variables is generally not good practice and at best makes interpretation of the coefficients troublesome; see this page for extensive discussion. You could, however, add a main effect of age along with this interaction, and that would be OK if justified by your understanding of the subject matter.

Second, it's not clear that this age:Location interaction really controls for an important issue at hand. If a linear model is valid (e.g., outcome linearly related to age), a model with that interaction and with age itself would correct for overall differences in outcome due to age, and differences between locations with respect to the influence of age, but it wouldn't correct for any influence of treatment on outcome that depends on age. I suspect that an age:treatment interaction would be even more of an issue than the age:Location interaction, particularly if clinicians at one location seem to be preferring one treatment over another based on age. So think carefully about what interactions make sense to include based on your understanding of factors that are contributing to outcome overall, and don't be focused in your model solely on the particular age mismatch between treatments at one location. Again, propensity score methods can be useful in dealing with this type of covariate imbalance to provide a model that can predicts how outcomes would differ between the two treatments if applied to the entire population.

Subsetting typically loses information by leaving informative cases out of the analysis. If a linear model including clinically relevant interactions (e.g., with Location or Treatment as needed) is well specified and calibrated then a combined model will take advantage of the greater number of cases to provide more precise estimates of the coefficients than will subsetting.

  • $\begingroup$ Thanks, but how is propensity scoring any better than comparing baseline characteristics across the groups? The later seems more simplistic and easier to interpret. I appreciate the answer, but the fundamental question still remains, if it is possible to include a term in a model that only applies to a specific factor of another group (see question). Thanks! $\endgroup$
    – CodeGuy
    Jan 28 '19 at 20:55
  • 1
    $\begingroup$ @CodeGuy Austin discusses pros and cons of regression adjustment versus propensity score methods. For regression adjustment to work you must have a well-specified model of how covariates relate to outcome; that's unlikely if you design the regression model based on covariate prevalence among groups, as you seem to propose. Propensity score methods are designed specifically to address your issue: taking observed groups that are unbalanced in covariates and matching or weighting cases to better approach a randomized study. $\endgroup$
    – EdM
    Jan 29 '19 at 17:07
  • $\begingroup$ Thanks for the info, but I would like to again reiterate the question which hasn't been answered: is it possible to include a term in a model that only applies to a specific factor of another group (see question) $\endgroup$
    – CodeGuy
    Jan 30 '19 at 14:44
  • $\begingroup$ @CodeGuy I've added a few paragraphs with respect to interaction terms in general and their application to your data set in particular. If you want to include interactions with age that's OK, but then you also need to include a main term for age. Again, include in your outcome model the variables that are reasonably related to outcome regardless of the particular age imbalance between treatments at one location, and consider propensity score methods like inverse treatment-propensity weighting for dealing with covariate imbalance. $\endgroup$
    – EdM
    Jan 30 '19 at 19:01
  • $\begingroup$ Thanks! Really helpful. Yes, I would include a main effect along with the interaction. I will certainly consider propensity scores too, need to read up more on that. $\endgroup$
    – CodeGuy
    Jan 31 '19 at 20:03

A couple of points:

  • I would advice against looking at the results of the t-tests to determine which variables and/or interactions to put in your model. These will suffer from multiple testing issues and may give you wrong impression on which variables seem to be related to your outcome. Moreover, to stably identify interaction terms with sufficient precision, you typically need large sample sizes. You have not mentioned how many subjects you have.
  • In mixed models a general model-building strategy is a follows:
    • You start with a “full” specification of your fixed effects. There you need to correct for variables that you think are associated with your outcome. There you should consider potential nonlinear terms for your continuous covariates, and interaction terms, taking into account the last sentence in the previous bullet point.
    • Then you build you random-effects structure. This is important because correct/efficient inference for the fixed effects depends on a correct/appropriate specification of the random-effects structure. In longitudinal studies, as it seems is the one you have, you start with random intercepts, and then you include random slopes, testing if they improve the fit, and moving to high order nonlinear terms if required.
    • When you have selected you random effects, you return to the fixed effects and see which variables seem to be associated with your outcome. You can start with omnibus tests to see if you can drop the interactions terms all together (using a bit higher significance level, e.g., 0.1 or 0.15), and doing likewise for the nonlinear terms. If these terms seem not to substantially improve the fit, you could drop them, and keep the full additive model in which you should not drop any other covariates.
  • If you are interested, you can have a look about these things in the note of my Repeated Measurements course.
  • $\begingroup$ Thanks for your answer. Regarding N, each TimePoint for each Treatment and each Location factor has about 50-100 data points. If a baseline characteristic is different, and if that baseline characteristic correlates with the outcome, to me it makes sense to include that to ensure that the effect is not driven by that baseline differences. I did complete all three steps you recommended, which then led me to this question haha. I'm hoping you may have some guidance more specifically about the question now that you have this extra bit of information :) $\endgroup$
    – CodeGuy
    Jan 24 '19 at 12:36
  • $\begingroup$ I think in my answer I did not suggest using t-tests for baseline characteristics. $\endgroup$ Jan 27 '19 at 7:32

Building a model by considering significance of predictors/their interactions is a generally poor policy that leads to poor properties of estimators and confidence intervals, a lack of generalizability/out of sample performance and so on. Whether something is "significant" or not does not really result in any meaningful properties of the predictor: it can still be incredibly important despite being "non-significant", "significance" can arise in a very spurious fashion and can be practically irrelevant. It matters much more whether there are considerations (theoretical/biological/based on how treatment decisions are made/etc.) why predictors/interactions should matter, whether the influence of a variable is so striking that you cannot ignore it and/or whether cross-validation shows improved performance (in terms of whatever you care about - the third criterion should hopefully cover the second one).

Another important question is whether this is confirmatory work (i.e. you want to publish the results of this and claim something e.g. with a p-value) or not. If it is, then you should really have a single pre-specified model, in which you perhaps specify a bunch of things you are not sure are needed, but which you leave in the model just in case they are needed. On the other hand, if you are trying to generate hypotheses/models/algorithms/whatever and then want to test them in a new experiment (or on a hold-out test set that you have made very sure to not look at while building your model) after you have decided on what you consider your most promising model, then you have a lot more freedom (but probably still want to do something like cross-validation/assessment on a validation set to not fool yourself).

Additionally, for comparing treatments something like propensity scores (as mentioned by @EdM) or structural equation models (if treatment decisions happen at multiple occasions) are well-known alternatives to regression adjustments.

I actually think it would be no problem to have a propensity score (or regression adjustment) that includes a location by age interaction, because if the treatment assignment mechanism simply includes these E.g. doctors in location a are more likely to assign a certain treatment to the elderly than doctors in the rest of the country, or if the middle aged people have worse outcomes in one part of the country than another, then those things are simply the sort of things that might be the case. What we care about is the treatment effect on top of that, so we try to adjust for these things. It becomes a different matter when we talk about treatment by location/age interactions. Those really lead to a different interpretation (if the interactions are not really small) that would imply that the treatment works different well by locations/age. That may be true and may be worth investigating, but it is almost a different question than the question for the average treatment effect adjusting for other factors.

  • $\begingroup$ I appreciate the answer, but the fundamental question still remains, if it is possible to include a term in a model that only applies to a specific factor of another group (see question). Thanks! $\endgroup$
    – CodeGuy
    Jan 28 '19 at 20:56
  • 1
    $\begingroup$ If you use propensity scores, this would potentially adjust for ti. I.e. the propensity score model would hopefully identify that in some locations age matters for what treatment you get (then they get a higher estimated prob. of getting that treatment, which then enters the propensity score matching). Similarly, with a regression adjustment, adding a loc.*age interaction in addition to main terms can potentially achieve the same thing. This assumes it results from the treatment assignment mechanism (i.e. they are not older, because treatment worked in those locations and they did not die). $\endgroup$
    – Björn
    Jan 29 '19 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.