2
$\begingroup$

Lets say two groups are compared. Subjects are randomly assigned to each group, then a treatment is given to half while a placebo is given to the other half. All aspects of the experiment (order of treatment, etc) are also randomized. The treatment group gets a much higher "score" than the placebo group on average, but not all in the treatment group score higher than all in the placebo group. When this occurs it would be usual to attribute the average difference to the treatment.

I would say two things against this:

1) The "weak causality" argument. The results are conditional on the exact environment under which data was collected. The treatment may not increase the scores if some seemingly minor aspect of the experiment is changed. In other words, the effect of the treatment is only "unmasked" if some other (unknown) criteria is/are met. The exact conditions of the experiment will never be repeated again.

2) The "randomization guarantees nothing argument". Randomization does not guarantee that the two groups are balanced on all important factors at baseline. It only makes it unlikely there is severe unbalance.

$\endgroup$
  • 3
    $\begingroup$ One thing it does do is largely remove a lot of alternative explanations. $\endgroup$ – Glen_b -Reinstate Monica Dec 11 '13 at 0:26
  • 1
    $\begingroup$ @Glen_b Is there some way of quantifying "largely remove"? This would seem to depend on sample size, the number of alternative explanations, and the size of the possible influence of those factors. The last two may be unknown. $\endgroup$ – Flask Dec 11 '13 at 0:50
  • 2
    $\begingroup$ Essentially, you seem to have the general idea. And yes, you can't really quantify with so many unknowns, but large samples and good randomization make it much harder for an alternative explanation to be tenable (you can use simulation as one way of assessing the extent to which it's possible) $\endgroup$ – Glen_b -Reinstate Monica Dec 11 '13 at 2:24
7
$\begingroup$

Your first point is about external validity. This is a problem with many experiments, especially on humans. This includes randomized trials. Just because a treatment works in the relatively controlled situation of a clinical trial (or other experiment) does not mean it will work elsewhere. However, this is not directly about causality.

Your second point is simply that we can never be certain of anything based only on statistical evidence. Indeed not. But 1) So what? Life is uncertain. Every science (whether based on statistical evidence or otherwise) makes mistakes. 2) We can increase the evidence of causality in various ways: a) Stronger statistical evidence (a large effect size is more convincing than a small one) b) Replication in other circumstances c) By figuring out the mechanism (e.g. we know a lot about how tobacco uses causes cancer).

"Proof" is something mathematicians get to; I don't think we data analysts can. But we can get overwhelming evidence.

$\endgroup$
  • 1
    $\begingroup$ No, you are using a different definition of "cause", admittedly a word that is extremely tricky! If you mean it to be "B follows A, always" then statistics has nothing to do. $\endgroup$ – Peter Flom - Reinstate Monica Dec 10 '13 at 23:55
  • 2
    $\begingroup$ Well, Judea Pearl wrote a whole book on it albeit one I find very hard to understand. Other definitions involve "producing an effect" but that doesn't help so much. It is a topic that has been very vexed. $\endgroup$ – Peter Flom - Reinstate Monica Dec 11 '13 at 0:25
  • 2
    $\begingroup$ @Flask: Mediated and moderated causality can entail prediction error. Smoking behaviors are certainly not the direct, unmediated, unmoderated causes of resultant cancer. Smoking can, on its own, cause changes in mediating factors (e.g., free radicals) that themselves cause cancer (or are at least closer to the final outcome of the causal chain), and moderating factors (e.g., genetic diathesis, or lack thereof) can prevent those mediating factors from resulting in cancer where applicable. Statisticians deal in direct, unmediated, unmoderated causes only rarely, if ever. $\endgroup$ – Nick Stauner Dec 11 '13 at 2:29
  • 1
    $\begingroup$ Just to add something basic that may be worth noting. The purpose of randomization is not to balance all unmeasured variables (obviously impossible) but rather to estimate their total effect + measurement error (usually with the error term) to see if they can plausibly account for the effect. The separation of measurement error and sampling error is not important here. $\endgroup$ – David Lane Feb 17 '17 at 0:17
  • 1
    $\begingroup$ @PeterFlom does not say it, but a reader might mistakenly infer from the first paragraph that non-experimental studies have an external-validity advantage. See Aronow and Samii (2015 AJPS) arguing that: "Causal effects estimated via multiple regression differentially weight each unit's contribution. The 'effective sample' that regression uses to generate the estimate may bear little resemblance to the population of interest," and therefore, "There is no general external validity basis for preferring multiple regression on representative samples over quasi-experimental or experimental methods." $\endgroup$ – Dr. Beeblebrox Jun 26 '17 at 15:35
2
$\begingroup$

It is widely believed that the answer to your question is "yes".

Because of your second argument "Randomization does not guarantee that the two groups are balanced on all important factors at baseline" I am not convinced either.

In more detail, I considered several quantitative models to validate the often-heard claim that "[Randomization] only makes it unlikely there is severe unbalance".

However, in general, the last sentence is wrong. You may find the results in "Randomization does not help much" (see http://xxx.tau.ac.il/abs/1311.4390).

$\endgroup$
  • 3
    $\begingroup$ I am not sure whether this answer adds value to @Peter Flom's answer. In an infinitely large sample, randomization in fact guarantees the independence of the treatment of any other confounding factor. Of course, in any finite sample, it might well be the case that the treatment will be correlated with some other observed or unobserved factor. See Imai et al., "Misunderstandings between experimentalists and observationalists about causal inference." Journal of the royal statistical society: series A, 171.2, especially section 5, for a nice discussion of this problem and what to do about it. $\endgroup$ – Julian Schuessler May 5 '14 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.