4
$\begingroup$

I want to match followers of some seed accounts with some random users on Twitter based on observed covariates using the coarsened exact matching method. My goal is to test whether following those seed Twitter accounts would change the outcome or not.

As for observed covariates, I can get, for example, # of tweets and # of followers. But these are post-treatment values, because the Twitter API gives me the current values of these variables. So can I use these post-treatment variables in my matching procedure, or I should only match on pre-treatment variables (e.g. # of tweets before the treatment, # of followers before the treatment, etc.)

Improve this question
$\endgroup$

3 Answers 3

Reset to default
6
$\begingroup$

I would suggest to match only on pre-treatment variables and not on post-treatment variables. If we match on post-treatment variables it is extremely plausible we induce selection bias in our sample and thus we get bogus insights. Selection bias can appear when one of the variables $C$ we use in matching is related to the treatment $A$ as well as the outcome of interest $Y$.

More formally and quoting Hernan & Robins directly: "selection bias can be defined as the bias resulting from conditioning on the common effect of two variables, one of which is either the treatment or associated with the treatment, and the other is either the outcome or associated with the outcome". Chapt. 8 "Selection Bias" from their upcoming book "Causal Inference". Therefore conditioning on post-treatment variables is a clear case of conditioning on common effects.

Notice that selection bias can be present not only because we have certain eligibility criteria (as those imposed by matching on a post-treatment variable $C$) but also through loss to follow-up; i.e. the drop-out rates (stopping the use of Twitter) might be different between the groups examined. The small paper on "Loss to follow-up" by Dettori offers a very brief, succinct introduction on this.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So, then what should I do with those post-treatment variables? How should I control for them? Say I count # of tweets before and after treatment. Then matched users according to # of tweets before treatment. Then how should I account for # of tweets after treatment? $\endgroup$
    – msmazh
    Commented Mar 24, 2018 at 19:24
  • 1
    $\begingroup$ We do not directly do anything with the post-treatment variables. That being said, they are useful, we can use these post-treatment variables as additional descriptors in the study or maybe use them to propose new hypotheses. We should be very careful not to use the post-treatment variables to immediately test new hypotheses though because that would amount to post-hoc theorising which is another problem in itself. $\endgroup$
    – usεr11852
    Commented Mar 25, 2018 at 1:26
  • $\begingroup$ Depending on the duration of the post period and when observations are measured, it's ambiguous if the outcome $Y$ is causing the post-treatment variable $X_{post}$ or $X_{post}$ is causing $Y$ - or both! In either case, the pre-treatment measure of the variable, $X_{pre}$, is also causally linked to $X_{post}$ so that $X_{post}$ is a collider which we want to avoid conditioning on. $\endgroup$
    – RobertF
    Commented Jan 27, 2023 at 21:05
3
$\begingroup$

As others have pointed out, adjusting for a post-treatment covariate can introduce bias in the estimated treatment effect. However, I would like to add that there are corner-case scenarios where adjusting for post-treatment covariate is acceptable. The basic idea is that if the covariate is unaffected of the treatment, the adjusting for it won't introduce bias. By adjusting for it, the precision of the estimated average treatment effect can then be improved. Furthermore, if you have confounding in the form of an unobserved pre-treatment covariate, then adjusting for a post-treatment covariate related to it can reduce the bias of your estimate under certain conditions. The details can be found in Rosenbaum (1984).

That said, I would be very reluctant to adjust for anything that is post-treatment, as the cases where such adjustment is beneficial are likely rare. As a general principle, avoid doing it. And if you plan on doing it, I strongly suggest reading the provided reference first.

References:

Rosenbaum, P. R. (1984). The consquences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society. Series A (General). 147(5): 656-666.

Improve this answer
$\endgroup$
2
$\begingroup$

Generally, your intuition is correct. Matching on post-treatment variables, i.e. variables that are determined after the treatment, carries the risk of biasing your estimates due to the bad control problem. This problem results from the fact that, loosely speaking, your control variables are influenced by the treatment and could be regarded as outcome variables themselves. For details on the bad control problem, see, for example, Angrist & Pischke (Mostly Harmless Econometrics, 2008).

Improve this answer
$\endgroup$
5
  • $\begingroup$ Selection bias is caused by conditioning on a variable $C$ that is a common effect of the treatment $A$ or a cause of the treatment $A$ and of the outcome $Y$ or a cause of the outcome $Y$. In the quoted sentence I refer to post-treatment variables (as the question of the OP does) which are a clear case of potentially common effect. Please check the book Causal Inference by Hernan & Robins, Chapt. 8. "Selection Bias" for more details. $\endgroup$
    – usεr11852
    Commented Mar 30, 2018 at 23:43
  • $\begingroup$ I added a direct quotation from H&R in my answer that I hope clarifies the point you raised. $\endgroup$
    – usεr11852
    Commented Mar 31, 2018 at 0:10
  • $\begingroup$ Right, I actually wanted to post a comment to your answer... The last sentence of your answer just sounds like selection bias is always a reason not to match. $\endgroup$
    – Murphy
    Commented Mar 31, 2018 at 6:58
  • $\begingroup$ Edited my answer. $\endgroup$
    – Murphy
    Commented Mar 31, 2018 at 6:59
  • $\begingroup$ No problem. But yes, if we are concerned with post-treatment variables it is very likely that matching on them will induce selection bias and should be avoided as a rule of thumb. In general, matching has its uses but one should also be wary of methods that discard otherwise valid points. Using more flexible models (eg. GAMs) and being an overall good Statistics citizen (e.g. checking model residuals, avoiding extrapolation, etc.) is more important (and productive). $\endgroup$
    – usεr11852
    Commented Mar 31, 2018 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.