4
$\begingroup$

The Issue: People attempt to draw causal inferences using many different statistical techniques (e.g. regression, propensity score matching, regression discontinuity, instrumental variables, etc.). One great way to learn about the strengths and weaknesses of different statistical techniques for causal inference is to compare them on the same data. Since randomized experiments are the so called "gold standard" for causal inference, they are obviously an excellent benchmark.

I have seen several studies of this last type, but I could only recall two. LaLonde's classic: "Evaluating the Econometric Evaluations of Training Programs with Experimental Data" and Aiken et. al. "Comparison of a Randomized and Two Quasi-Experimental Designs in a Single Outcome Evaluation: Efficacy of a University-Level Remedial Writing Program."

Do you know of other examples of this type of study?

$\endgroup$
  • 1
    $\begingroup$ I'd go back to Mostly Harmless Econometrics -- IIRC the authors use a lot of these as examples. $\endgroup$ – generic_user Mar 17 '15 at 22:38
3
$\begingroup$

The type of study you are referring to is called a within-study comparison. An early example that produced a lot of discussion is Dehejia and Wahba (1999; JASA) using Lalonde's (1986) NSW data in which they compared the results based on PSA to the randomized experimental benchmark. The Lalonde data set is now included in PSA packages in R such as Matching and twang, for example. There was an ongoing workshop at Northwestern (not sure if they still do it) that has an archived website with a reference list you will find useful (link).

One interesting example is the 2008 JASA paper by Shadish, Clark, and Steiner in which they randomized participants to be in either an observational study or a randomized experiment and then used the results from the randomized experiment as a benchmark, as you say. The more typical design is three arm (randomized treatment gp, randomized comparison group, observational comparison group). Shadish, Clark, and Steiner's design was four arm (randomized treatment gp, randomized comparison group, observational treatment gp, observational comparison group).

$\endgroup$
1
$\begingroup$

In the medicine, the most recent and comprehensive work I'm aware of has been done by OMOP (the Observational Medical Outcomes Partnership). You'll find a lot of relevant research on their publications page, and I think the review paper, 'A systematic statistical approach to evaluating evidence from observational studies', gives a good overview of the project and its findings.

$\endgroup$
1
$\begingroup$

Angus Deaton, the latest Economics Nobel Laureate, is interviewed in this link regarding his thoughts on RCTs as the gold standard. He's quite refreshingly skeptical pointing to, among other things, the typically small sample sizes in RCTs vs the nationally projectable estimates available from observational studies and concluding that, "I don’t see a difference in terms of quality of evidence or usefulness. There are bad studies of all sorts."

https://medium.com/@timothyogden/experimental-conversations-angus-deaton-b2f768dffd57#.t41xnnnd5

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.