3
$\begingroup$

I am analyzing a dataset where pre/post measurements are made (before/after a surgery) on patients who can belong to two groups, and we are interested in how group membership - let's call it G - affects the impact of the surgery. (It is a nonrandomized observational study, but let's put other confounders aside now for simplicity.)

I am no expert in such situations, so I read a bit, and it seems that percentage change is a bad idea. As this is not a randomized trial, it is also not a question that we can not neglect the baseline measurement. Thus, we are left with two options:

lm( I(Post-Pre) ~ G, data = RawData )

(Intercept)            G  
   -0.02563      1.04204 

i.e. change score, or

lm( Post ~ Pre + G, data = RawData )

(Intercept)          Pre            G  
  -0.003125     0.006014     0.973866  

i.e. baseline adjusted as a covariate. (That is, ANCOVA, if I am not mistaken with the terminology.)

What do they mean, and how they are related to each other is discussed in the literature in great detail. My understanding is that generally speaking the latter is the better approach. Hope I am right up to this point.

Now, what I don't understand is how paired $t$ test fits into this picture. That is, how

lme( Meas ~ PrePost * G, random = ~1|id,
          data = reshape( RawData, varying = c( "Pre", "Post" ), v.names = "Meas",
          timevar = "PrePost", direction = "long" ) )

(Intercept)     PrePost           G   PrePost:G 
 0.02264409 -0.02563254 -0.06858446  1.04203755 

Random effects:
 Formula: ~1 | id
        (Intercept) Residual
StdDev:   0.0802519 1.034537

relates to the above procedures...? (I called it "paired $t$ test-ish" because lme( Meas ~ PrePost, random = ~1|id ) would have been equivalent to the usual paired $t$-test, with no group membership.)

I don't think that it is equivalent to any of the above approaches, and it might be completely pointless (I have the feeling that it measures something different), but I simply don't see what it means...

(The above examples pertain to set.seed( 1 ) and RawData <- data.frame( Pre = rnorm( 1000 ), Post = rep( 0:1, each = 500 ) + rnorm( 1000 ), G = rep( 0:1, each = 500 ), id = 1:1000 ).)

$\endgroup$
  • $\begingroup$ +1. Would you post the output of all three options so that we could see how different/similar they are? $\endgroup$ – amoeba May 22 '17 at 14:22
  • $\begingroup$ @amoeba: Sure! (Although I intended the code only as a clear demonstration of the concept, not as a concrete numerical example.) But now I extended it. $\endgroup$ – Tamas Ferenci May 22 '17 at 14:52
  • $\begingroup$ I don't really know R but I suspect that there is something wrong with your lme call or perhaps with the reshape inside it. The output does not make sense. $\endgroup$ – amoeba May 22 '17 at 15:00
  • $\begingroup$ @amoeba: Why do you think it doesn't make sense? It just reports an intercept and the coefficients for the two right hand side variables (just as in the ANCOVA case). $\endgroup$ – Tamas Ferenci May 22 '17 at 15:12
  • 1
    $\begingroup$ @amoeba : Broadly speaking: what are the pros/cons of these three approaches? When should we use them (even if just "generally speaking")...? I understand that change score is typically not a good idea, so the question rather pertains to the ANCOVA approach vs. mixed-effects approach. More specifically: how are these methods related, is there any mathematical connection between them? Again, not between ANCOVA and change score, because I know that they're extensively compared in the literature, but between these methods and the mixed-effects approach. $\endgroup$ – Tamas Ferenci May 24 '17 at 15:47
3
$\begingroup$

The paired t-test is best applied to a crossover study so that order effects can cancel out. For observational pre-post comparisons the design is very weak.

The ANCOVA on the raw response is preferred because the slope on pre might not be 1.0 and the difference post-pre may not be ordinal if post and pre are ordinal but not interval scaled. Ordinal regression is a general solution, and must be done on raw values.

Your ANCOVA example is comparing 2 independent groups but the paired t is testing a completely different within-group hypothesis. I'm not as clear on the random effects model. You'll see old discussions of this in the statistics literature under the phrase "recovery of intra-block information".

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you! I think here lies the key: "Your ANCOVA example is comparing 2 independent groups but the paired t is testing a completely different within-group hypothesis" Could you please elaborate this a bit...? Right now I believe that the 3. option (paired t test like, random effects) is an extension of the change scores approach - check the coefficients! - so they're very strongly related, but I can't prove it... $\endgroup$ – Tamas Ferenci May 24 '17 at 9:09
  • $\begingroup$ I was just referring to the case where the dataset contains one row per subject and there is a grouping variable. Then the grouping variable contrasts two independent groups of subjects. $\endgroup$ – Frank Harrell May 24 '17 at 12:30
1
$\begingroup$

I think the complaints brought up by the Vanderbilt site are valid ones, but they can't be applied universally without knowledge of the question. They mention specific graphical tests which can be used to assess assumptions about a magnitude of effect. For instance, in HIV rebound patients, administration of antiretroviral therapy (ART) has a demonstrable geometric mean effect. That's because when the virus is more prolific, the PK dynamics are such that the virus interacts at a higher rate with the ART. Issues with interpretation behoove us to think more carefully about our interpretation than to seek in vain an omnibus that permits us to be loose with interpretation.

You bring up two separate points: ANCOVA vs paired differences and percent change as the scale of effect. Modeling change as a percentage involves performing a log transform of the outcome, re-exponentiating the coefficients obtained from the linear model, and interpreting them as a percentage change. A paired t-test is a special case of a linear model without log-transform where you actually model an offset:

\begin{equation} E[Y_1 | Y_0, X] = \beta_0 + Y_0 + \beta_1 X \end{equation}

$\beta_1$ is the intervention effect as a mean difference obtained from a paired t-test, or a univariate t-test based on pairwise differences in $Y_1, Y_0$. Note the term $Y_0$ does not have a coefficient.

The ANCOVA is seen as a better model because it is a more flexible parametric model which incorporates a coefficient for the referent/baseline term.

\begin{equation} E[Y_1 | Y_0, X] = \beta_0 + \gamma Y_0 + \beta_1 X \end{equation}

$\beta_1$ still has the same interpretation: the expected difference in outcome comparing individuals who receive the intervention to those who don't conditional upon baseline/referent value.

To model percent change in an ANCOVA design, you apply log transform of the outcome.

\begin{equation} E[\log Y_1 | Y_0, X] = \beta_0 + \gamma Y_0 + \beta_1 X \end{equation}

And $\exp(\beta_1)$ is interpreted as a percent difference in outcome comparing those who receive intervention to those who don't conditional on their baseline/referent value.

The issue of homogeneity which is addressed in the Vanderbilt site is also worth mention. Suppose you are interested in modeling percent change as with our final log transformed ANCOVA model. Suppose further, there is a truncation of treatment effect for a range of values, like with our HIV patients receiving ARTs. If the viral load drops below a threshold the devices cannot detect its presence. In many data analyses, analysts impute the minimum (a pragmatic and inefficient solution). This would, by assumption, violate the predicted trend. However, we can view the model as summarizing those data in spite of those differences, or any other subgroup differences, when we interpret the $\beta_1$ or $\exp \beta_1$ as the Average Causal Effect (or ACE). This is an issue of interpretation that considers the whole population rather than what individual outcomes are, which is often a follow-up question requiring much more extensive design and investigation.

The mixed model is a bit confusing since you haven't defined terms. Nonetheless, a mixed approach is the way to go if there are more than one post measurement. Why would you assess more than one post measurement? Measurement error: blood pressure? different every day, viral load? different every day, weight? you get the picture. By design if an outcome can be measured cheaply but has low reliability, an efficient design might specify multiple measurements as a way of reducing intraindividual variability. However, in doing this, we now have correlated measures. Introducing a random intercept further reduces intraindividual variability, and hence the overall variability of the effect estimate, by inducing an exchangeable correlation structure where each repeated measure within an individual has a fixed, constant, positive correlation. This design feature updates ANCOVA to be a "repeated measures ANCOVA".

As a note your R notation is wrong. Note in my describing the above design: we incorporate a baseline measure as a covariate in the ANCOVA design or an offset in the paired design. In your reshape dataset, the pre and post measures are taken as exogenous measures which are predicted by G which I presume is the intervention design. Baseline measures are NOT predicted by this value, so that model is wrong. "Repeated measures" in a pre-post design means you have more than one measure of the outcome.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the detailed answer! I'm still trying to understand every part, in the meantime let me reflect on two remarks: "The mixed model is a bit confusing since you haven't defined terms. Nonetheless, a mixed approach is the way to go if there are more than one post measurement." No! We have exactly one measurement before, and one after. The reason why I used random effects model is purely syntactical: I took the example of the paired t test, and I "mechanically" extended it with a covariate (G). $\endgroup$ – Tamas Ferenci May 24 '17 at 8:58
  • $\begingroup$ "or an offset in the paired design" Okay, I understand this solution, but my question still stands: what happens if you do it the way I did...? (Please note that I edited this part, the original was not what I intended, sorry!) Does it have a meaning? If so, what? What does it test? How is it related to the previous examples? $\endgroup$ – Tamas Ferenci May 24 '17 at 9:01
  • $\begingroup$ "In your reshape dataset, the pre and post measures are taken as exogenous measures which are predicted by G which I presume is the intervention design. Baseline measures are NOT predicted by this value, so that model is wrong." I'm not perfectly sure that I understood you, however: why is this necessarily wrong? This was not a randomized trial, rather an observation, so baselines may very well be different between groups! I.e. I can predict even the baseline value with G. $\endgroup$ – Tamas Ferenci May 24 '17 at 9:03
  • $\begingroup$ @TamasFerenci what is "mechanical extension" of a paired t-test? Do you mean that you regressed "pre" as a covariate and "g" as a group indicator on "post" as an outcome in a linear model and conducted inference on g? $\endgroup$ – AdamO May 24 '17 at 15:57
  • $\begingroup$ @TamasFerenci can you explain the nature of the "group"? Is this an exposure (a term I'll use here on)? Were participants exposed before the "pre" measurement was taken? I presume not, since it would not be a pre-post design otherwise. If not, is the "exposure" value coded to 0 for all rows in which the outcome is a pre-measurement and coded to 1 only for the rows where the individual is exposed? $\endgroup$ – AdamO May 24 '17 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.