4
$\begingroup$

I have a mixed between-within design, with three groups and Pre (T1) and Post (T2) measurements. I'm hesitating on the right statistical analyses to do, but I would like to compare each group to the other two separately while taking into account the baseline measurement. I would like to run multiple regressions with planned contrasts. I came up with 4 options:

Note: this question is not about omnibus tests such as ANOVA, ANCOVA, etc.

(a) Planned contrasts between groups at Post (but does not take T1 into account, so just as a reference)

# Model in R:
lm(T2 ~ Group)

(b) Planned contrasts between groups on change score (Post - Pre)

# Model in R:
delta <- T2-T1
lm(delta ~ Group)

(c) Planned contrasts between groups at Post, while controlling for Pre (in the regression model)

# Model in R:
lm(T2 ~ Group + T1)

(d) Planned contrasts between groups on change score (Post - Pre), while controlling for Pre (in the regression model)

# Model in R:
delta <- T2-T1
lm(delta ~ Group + T1)

Furthermore, according to this guy, it is also important to include interactions with covariates in the model to prevent Type 1 errors, so the models (c) and (d) would actually look like this:

lm(T2 ~ Group + T1 + Group:T1)
# Or
lm(delta ~ Group + T1 + Group:T1)

Related questions: 1, 2, 3.

Question

Which option is best? Is option (d) redundant given that it controls for time 1 in the model AND also uses delta (T2-T1) as a dependent variable? On the contrary, are there any additional benefits to doing both?

$\endgroup$
1
  • $\begingroup$ Thank you @mkt-ReinstateMonica, I think the thread is definitely relevant and that, fundamentally, the accepted answer does answer my question (that using T1 as covariate is better than using T2-T1). I still think that question is different to the extent that it focuses on ANOVA, so I didn't know if it applied to my situation of pair-wise comparisons (planned contrasts). The three options proposed by that OP are not exactly the same as the four options I describe here (with accompanying R code for clarity). Thanks for sharing! $\endgroup$ – RemPsyc Sep 28 '20 at 17:03
4
$\begingroup$

The options under (d) are wrong, as a change score is associated with the baseline value. See this page, for example.

Otherwise, it depends on what you mean by "taking into account the baseline measurement." You already note that option (a) doesn't do that at all.

Option (b) looks only at the change from baseline as a function of Group. Based on your knowledge of the subject matter, do you think that is an adequate way to take the baseline into account? The advantage is all that you estimate is 3 parameter values.

Option (c) allows for a slope in the relationship between T2 and T1, with the same slope for all Groups. (One could think of option (b) as forcing that slope to be 1 for all Groups.) But adding the slope to the model means you're now up to 4 parameter values to estimate.

You could extend option (c) to include an interaction between Group and T1, allowing for different slopes among the Groups. That's a more complicated model, now with 6 parameter values to estimate by my count.

So there is no clear answer about which is "best." More complicated models can capture more details about what's going on. The extra number of parameter values estimated from the data, however, can diminish the power to document truly significant relationships. A more complicated model and also lead to overfitting, building a model that fits your data set well but doesn't generalize to the underlying population. That can be a particular problem with small data sets. In many linear regression studies you typically want to have 10-20 cases per parameter estimated by the model, so if you have few cases you might need to restrict yourself to simpler models.

Added in response to comments:

This page and its links extensively discuss change scores, Option (b), versus regression of final values against initial values and a group indicator, Option (c). Allison provides a thorough comparison. As he says (page 106):

It is unrealistic to expect either model to be the best in all situations; indeed, I shall argue that each of these models has its appropriate sphere of application.

You will note, however, that Allison's arguments in favor of the change score in some circumstances are based on Option (b) without including the baseline value T1 as a predictor as Option (d) envisions. Consistent with that, Glymour et al report:

... in many plausible situations, baseline adjustment induces a spurious statistical association between education and change in cognitive score... In some cases, change-score analyses without baseline adjustment provide unbiased causal effect estimates when baseline-adjusted estimates are biased.

Although Clifton & Clifton argue for including the baseline as a covariate when change scores are an outcome, they provide many cautions such as:

Using change score as outcome has undesirable implications... By contrast, using post scores is always valid and never misleading.

Both those arguments, for including baseline as a covariate and that "using post scores is always valid," seem to disagree with Allison's presentation in favor or change scores in some circumstances, as I understand it.

Alternate approaches

One might avoid some of these arguments with alternate modeling approaches.

In some fields of study, errors tend to be proportional to observed values and effects are multiplicative rather than additive. If that's the case in your field of study, working with log-transformed values of T1 and T2 with a model like Option (c) provides a coefficient for T1 that expresses the fractional change in T2 per fractional change in T1, which is maybe even easier to explain than what you would get from the corresponding analysis of untransformed values.

A mixed model that includes both T1 and T2 values as outcomes, with an indicator of the time of observation as a predictor, would have the advantage of putting T1 and T2 on equal footing. The fixed-effects regression approach in Option (c) implicitly assumes that T1 is known precisely and that all error is associated with T2. A mixed model with a random intercept for each individual could provide a way to "[take] into account the baseline measurement" that shares information from both T1 and T2 to get a potentially more reliable estimate of the true baseline condition rather than the particular observed baseline value.

Looking over all of these different approaches, I think that this still comes down to what I said in the second paragraph:

it depends on what you mean by "taking into account the baseline measurement."

You have to use your knowledge of the subject matter to decide which accounting is most appropriate.

$\endgroup$
4
  • 2
    $\begingroup$ (+1) I agree that option (d) should be avoided. Nonetheless it's important to note that the major difference in the results will be that the coefficient of T1 will be 1 less than in option (c) (yes, the $t$-value and $p$-value will change as well). The coefficient for Group (as well as its $p$-value) will be exactly identical, however. $\endgroup$ – COOLSerdash Sep 25 '20 at 19:19
  • $\begingroup$ Thank you! Just for clarification, on another thread, Frank Harrell states that "If one wants to use a change score... it is mandatory to include the baseline on the right hand side also. This is because the slope of the baseline against the final weight may not be one." $\endgroup$ – RemPsyc Sep 25 '20 at 22:39
  • $\begingroup$ Furthermore, Clifton & Clifton (2019) argue that "using the change score as the outcome measure does not address the problem of regression to the mean, nor does it take account of the baseline imbalance. Whether the outcome is change score or post score, one should always adjust for baseline... otherwise, the estimated treat effect may be biased." Am I misunderstanding these authors? Do they conflict with Glymour et al. (2005)? $\endgroup$ – RemPsyc Sep 25 '20 at 22:40
  • 2
    $\begingroup$ @RemPsyc Harrell's comment critically starts: "If one wants to use a change score (why?)..." In his answer he says: "The overall preferred method is to adjust for baseline and to model the response variable, not computing the change." That argues for option (c) or its interaction variant, among your options. I will discuss further in an addition to the answer when I get a chance. $\endgroup$ – EdM Sep 26 '20 at 13:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.