2
$\begingroup$

I want to match followers of some seed accounts with some random users on Twitter based on observed covariates using the coarsened exact matching method. My goal is to test whether following those seed Twitter accounts would change the outcome or not.

As for observed covariates, I can get, for example, # of tweets and # of followers. But these are post-treatment values, because the Twitter API gives me the current values of these variables. So can I use these post-treatment variables in my matching procedure, or I should only match on pre-treatment variables (e.g. # of tweets before the treatment, # of followers before the treatment, etc.)

$\endgroup$
5
$\begingroup$

I would suggest to match only on pre-treatment variables and not on post-treatment variables. If we match on post-treatment variables it is extremely plausible we induce selection bias in our sample and thus we get bogus insights. Selection bias can appear when one of the variables $C$ we use in matching is related to the treatment $A$ as well as the outcome of interest $Y$.

More formally and quoting Hernan & Robins directly: "selection bias can be defined as the bias resulting from conditioning on the common effect of two variables, one of which is either the treatment or associated with the treatment, and the other is either the outcome or associated with the outcome". Chapt. 8 "Selection Bias" from their upcoming book "Causal Inference". Therefore conditioning on post-treatment variables is a clear case of conditioning on common effects.

Notice that selection bias can be present not only because we have certain eligibility criteria (as those imposed by matching on a post-treatment variable $C$) but also through loss to follow-up; i.e. the drop-out rates (stopping the use of Twitter) might be different between the groups examined. The small paper on "Loss to follow-up" by Dettori offers a very brief, succinct introduction on this.

$\endgroup$
2
  • $\begingroup$ So, then what should I do with those post-treatment variables? How should I control for them? Say I count # of tweets before and after treatment. Then matched users according to # of tweets before treatment. Then how should I account for # of tweets after treatment? $\endgroup$ – msmazh Mar 24 '18 at 19:24
  • 1
    $\begingroup$ We do not directly do anything with the post-treatment variables. That being said, they are useful, we can use these post-treatment variables as additional descriptors in the study or maybe use them to propose new hypotheses. We should be very careful not to use the post-treatment variables to immediately test new hypotheses though because that would amount to post-hoc theorising which is another problem in itself. $\endgroup$ – usεr11852 Mar 25 '18 at 1:26
2
$\begingroup$

As others have pointed out, adjusting for a post-treatment covariate can introduce bias in the estimated treatment effect. However, I would like to add that there are corner-case scenarios where adjusting for post-treatment covariate is acceptable. The basic idea is that if the covariate is unaffected of the treatment, the adjusting for it won't introduce bias. By adjusting for it, the precision of the estimated average treatment effect can then be improved. Furthermore, if you have confounding in the form of an unobserved pre-treatment covariate, then adjusting for a post-treatment covariate related to it can reduce the bias of your estimate under certain conditions. The details can be found in Rosenbaum (1984).

That said, I would be very reluctant to adjust for anything that is post-treatment, as the cases where such adjustment is beneficial are likely rare. As a general principle, avoid doing it. And if you plan on doing it, I strongly suggest reading the provided reference first.

References:

Rosenbaum, P. R. (1984). The consquences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society. Series A (General). 147(5): 656-666.

$\endgroup$
1
$\begingroup$

Generally, your intuition is correct. Matching on post-treatment variables, i.e. variables that are determined after the treatment, carries the risk of biasing your estimates due to the bad control problem. This problem results from the fact that, loosely speaking, your control variables are influenced by the treatment and could be regarded as outcome variables themselves. For details on the bad control problem, see, for example, Angrist & Pischke (Mostly Harmless Econometrics, 2008).

$\endgroup$
5
  • $\begingroup$ Selection bias is caused by conditioning on a variable $C$ that is a common effect of the treatment $A$ or a cause of the treatment $A$ and of the outcome $Y$ or a cause of the outcome $Y$. In the quoted sentence I refer to post-treatment variables (as the question of the OP does) which are a clear case of potentially common effect. Please check the book Causal Inference by Hernan & Robins, Chapt. 8. "Selection Bias" for more details. $\endgroup$ – usεr11852 Mar 30 '18 at 23:43
  • $\begingroup$ I added a direct quotation from H&R in my answer that I hope clarifies the point you raised. $\endgroup$ – usεr11852 Mar 31 '18 at 0:10
  • $\begingroup$ Right, I actually wanted to post a comment to your answer... The last sentence of your answer just sounds like selection bias is always a reason not to match. $\endgroup$ – Murphy Mar 31 '18 at 6:58
  • $\begingroup$ Edited my answer. $\endgroup$ – Murphy Mar 31 '18 at 6:59
  • $\begingroup$ No problem. But yes, if we are concerned with post-treatment variables it is very likely that matching on them will induce selection bias and should be avoided as a rule of thumb. In general, matching has its uses but one should also be wary of methods that discard otherwise valid points. Using more flexible models (eg. GAMs) and being an overall good Statistics citizen (e.g. checking model residuals, avoiding extrapolation, etc.) is more important (and productive). $\endgroup$ – usεr11852 Mar 31 '18 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.