1
$\begingroup$

Suppose you are interested in how an introduction of some X causes a change in some metric Y in a population. Normally, you would random sample an experiment and control group, introduce the X into the experiment group, and after a while, could measure the difference in Y, compute a confidence interval ect, and determine if X probably caused a change in Y.

However, suppose the sampling wasn't done randomly. As such, you realize that before the experiment took place, C had a different average value of y then E. How do you adjust the statistics to compute if X had a significant change in y. Would you just minus the bias in y before the experiment from the result value of y after the experiment? I'm interested in how this would be derived statistically if correct. Thanks for the time.

$\endgroup$
3
$\begingroup$

You can consider propensity scores. Roughly, you figure out which variables influence the choice to enter one group or the other. If you condition on these variables, the resulting subgroups act as though (IN THEORY!) there was random assignment to treatments. For example, suppose we get a group of patients who have taken new treatment X vs another group that took control Y. We find out that the only thing that changes peoples' propensity to take treatment X is how sick they are. So we stratify the patients into a sick group and a not-so-sick group. Then WITHIN each of those groups, choice whether to take X vs Y is completely random and so the treatments can be compared without bias if there is conditioning on the 'sickness level'. I think Donald Rubin worked a lot with propensity scores.

If you are not a fan of propensity scores, then you are stuck with the usual set of procedures to try to wring evidence of causality from observations studies: See the Bradford Hill criteria, for instance https://en.wikipedia.org/wiki/Bradford_Hill_criteria

$\endgroup$
2
$\begingroup$

This is a standard issue in observational studies, the literature is voluminous and, as usual, contradictory. Gary King, Harvard political scientist, recently (August 2015) posted a paper titled Why Propensity Scores Should Not Be Used for Matching in observational studies. His case is careful, convincing and demonstrates that matching based on minimizing the Mahalanobis distance is much more accurate than propensity score matching, based on the same criteria. Then, on Friday, Sept 11th he gave an ICM talk that, in 60 minutes, reviews the key findings from the paper, accompanied with some great graphical evidence. You can check out the archived webinar at the link below. Short of writing your code for you, it's a very clear "how-to."

http://gking.harvard.edu/files/gking/files/psnot.pdf

http://www.methods-colloquium.com/#!Gary-King-Why-Propensity-Scores-Should-Not-Be-Used-for-Matching/clv6/55f31b8b0cf20cc524a66366

http://www.methods-colloquium.com/

$\endgroup$
1
$\begingroup$

This is the standard issue faced in observational studies and I hope I am giving an okay starting point for googling the extensive literature on this topic. If you have observed all variables that might influence group assignment (and the outcome) then options such as stratification by propensity score strata (e.g. deciles) can work well. Since in practice you cannot be 100% sure you have data on all possible confounders, one needs to also think about whether one had at least the most important ones and whether ommitted ones might still be able to overturn the analysis results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.