In clinical trials, treatment and non-treatment groups go through random allocation, in the hope that characteristics of both groups are similar at baseline. If there are any baseline differences they can be added as covariates to adjust for them, eg in ANCOVA.

When comparing treatment effect in observational studies, where two groups differ at baseline, I understand this approach is not suitable. Propensity score matching is a popular approach. My questions are:

  1. Why can't observational studies adjust for baseline differences by adding them as covariates?
  2. What are other alternatives to propensity score matching and their benefits?
  3. Imagine a flipped example: In an observational study of N number of people on treatment X, I want to compare response to treatment X between alcohol drinkers and non-drinkers. I hypothesise that drinkers will respond less well. I know that drinkers have worse disease at baseline, so this difference needs to be adjusted. I can pretend that drinking is the "treatment" and use propensity score to match drinkers and non-drinkers for baseline disease severity. However this analogy seems to me not completely correct, as drinkers will have been drinking long before they started treatment (c.f. in an observational study comparing treatment effect, the individual will have worse disease before they start treatment). Therefore is PSM unsuitable, and if so what are the alternatives?

1) They can! The problem is that if the functional form of the relationship between the covariates and the outcome is not exactly as you specify in the regression/ANCOVA, then you arrive at a biased and inconsistent estimate of the treatment effect. This matter is discussed in many articles, but the key articles for this are Schafer & Kang (2008) and Ho, Imai, King, & Stuart (2007). Another issue (mentioned in Schafer & Kang) is that if there is effect modification (the effect of treatment differs across levels of a covariate), but you only care about a marginal treatment effect (i.e., over an entire population receiving a policy change), then the effect cannot be estimated with regression (i.e., because if you include an interaction term with treatment, then the base treatment coefficient will not equal the causal effect).

2) Other approaches include inverse probability weighting, stratification, and other non-propensity-score-based matching techniques. Inverse probability weighting uses propensity scores to compute weights which are applied to your data set to create a pseudosample in which the covariates are balanced; the weights are then used in WLS regression to get a treatment effect. Stratification by the propensity score involves estimating a propensity score and creating strata based on quantiles of the propensity score. Under ideal conditions, there will be covariate balance within each stratum, and you can compute and then average stratum-specific causal effects to arrive at an average effect estimate. There are also other matching methods that don't involve propensity scores: Mahalanobis distance matching, genetic matching, and coarsened exact matching come to mind.

3) There is a technique called marginal structure models (MSM) with time-varying treatments that might be of use. Essentially, you have three timepoints: at t1, people are "assigned" to being a drinker or not; at t2, people are "assigned" to being in treatment or not, and this assignment depends on treatment assignment at t1. Finally there is the outcome. Using MSM, you can estimate counterfactual means for the outcome for a given treatment "history". You are probably interested in all four combinations resulting from a complete cross of drinking and treatment. Then you can compare those means however you like to arrive at the effect of interest. For example, for the effect of treatment in those who are drinkers, you could calculate the difference in the means between the those who received treatment history (drinker, treatment) and those who received treatment history (drink, control).

  • $\begingroup$ Thank you so much @Noah, that is very helpful I will have a thorough read of your recommended references. I've struggled (as a non-statistician) to get to grips with MSM, and I've not had much luck in finding a gentle introduction; in fact I believe I was awarded a tumbleweed badge for a question on MSM which received no attention! Regarding potential alternatives to PSM - this paper refers to this method I wonder if you've come across this before? $\endgroup$ – bobmcpop Apr 30 '17 at 14:37
  • 1
    $\begingroup$ I had not come across that paper or that method, and it seems quite complicated to understand on a first read through. I think it may be aimed at more longitudinal data than what you proposed. I do think MSM is what you want. I agree that there aren't any gentle introductions, which is problematic. Maybe I'll try to write one someday :) $\endgroup$ – Noah Apr 30 '17 at 18:24
  • $\begingroup$ Please let me know if you ever write that paper! Also, I realised my original question wasn't very clear: essentially I want to do exactly what the Cuirea paper did, except no one (not even our statisticians) knew exactly how the Harrison methods works. The population are all on the treatment, and I'm wanting to compare treatment effect between drinkers and non-drinkers, bearing in mind that drinkers have worse disease at baseline. I will try and read up on MSM :) $\endgroup$ – bobmcpop Apr 30 '17 at 20:20
  • $\begingroup$ Regarding Q/A 1: Please see point 5 in this journal guideline. Do you think they are overzealous in banning all baseline adjustment? $\endgroup$ – bobmcpop May 4 '17 at 6:46
  • 1
    $\begingroup$ I believe in that document they mean baseline measurements of the outcome variable, not all baselines covariates. I'm not certain, but I think if you use propensity scores, this issue doesn't apply. I would ask someone with more knowledge, though. $\endgroup$ – Noah May 4 '17 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.