3
$\begingroup$

Lets say two groups are compared. Subjects are randomly assigned to each group, then a treatment is given to half while a placebo is given to the other half. All aspects of the experiment (order of treatment, etc) are also randomized. The treatment group gets a much higher "score" than the placebo group on average, but not all in the treatment group score higher than all in the placebo group. When this occurs it would be usual to attribute the average difference to the treatment.

I would say two things against this:

1) The "weak causality" argument. The results are conditional on the exact environment under which data was collected. The treatment may not increase the scores if some seemingly minor aspect of the experiment is changed. In other words, the effect of the treatment is only "unmasked" if some other (unknown) criteria is/are met. The exact conditions of the experiment will never be repeated again.

2) The "randomization guarantees nothing argument". Randomization does not guarantee that the two groups are balanced on all important factors at baseline. It only makes it unlikely there is severe unbalance.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ One thing it does do is largely remove a lot of alternative explanations. $\endgroup$
    – Glen_b
    Commented Dec 11, 2013 at 0:26
  • 1
    $\begingroup$ @Glen_b Is there some way of quantifying "largely remove"? This would seem to depend on sample size, the number of alternative explanations, and the size of the possible influence of those factors. The last two may be unknown. $\endgroup$
    – Flask
    Commented Dec 11, 2013 at 0:50
  • 2
    $\begingroup$ Essentially, you seem to have the general idea. And yes, you can't really quantify with so many unknowns, but large samples and good randomization make it much harder for an alternative explanation to be tenable (you can use simulation as one way of assessing the extent to which it's possible) $\endgroup$
    – Glen_b
    Commented Dec 11, 2013 at 2:24

2 Answers 2

Reset to default
9
$\begingroup$

Your first point is about external validity. This is a problem with many experiments, especially on humans. This includes randomized trials. Just because a treatment works in the relatively controlled situation of a clinical trial (or other experiment) does not mean it will work elsewhere. However, this is not directly about causality.

Your second point is simply that we can never be certain of anything based only on statistical evidence. Indeed not. But 1) So what? Life is uncertain. Every science (whether based on statistical evidence or otherwise) makes mistakes. 2) We can increase the evidence of causality in various ways: a) Stronger statistical evidence (a large effect size is more convincing than a small one) b) Replication in other circumstances c) By figuring out the mechanism (e.g. we know a lot about how tobacco uses causes cancer).

"Proof" is something mathematicians get to; I don't think we data analysts can. But we can get overwhelming evidence.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ No, you are using a different definition of "cause", admittedly a word that is extremely tricky! If you mean it to be "B follows A, always" then statistics has nothing to do. $\endgroup$
    – Peter Flom
    Commented Dec 10, 2013 at 23:55
  • 2
    $\begingroup$ Well, Judea Pearl wrote a whole book on it albeit one I find very hard to understand. Other definitions involve "producing an effect" but that doesn't help so much. It is a topic that has been very vexed. $\endgroup$
    – Peter Flom
    Commented Dec 11, 2013 at 0:25
  • 2
    $\begingroup$ @Flask: Mediated and moderated causality can entail prediction error. Smoking behaviors are certainly not the direct, unmediated, unmoderated causes of resultant cancer. Smoking can, on its own, cause changes in mediating factors (e.g., free radicals) that themselves cause cancer (or are at least closer to the final outcome of the causal chain), and moderating factors (e.g., genetic diathesis, or lack thereof) can prevent those mediating factors from resulting in cancer where applicable. Statisticians deal in direct, unmediated, unmoderated causes only rarely, if ever. $\endgroup$
    – Nick Stauner
    Commented Dec 11, 2013 at 2:29
  • 1
    $\begingroup$ Just to add something basic that may be worth noting. The purpose of randomization is not to balance all unmeasured variables (obviously impossible) but rather to estimate their total effect + measurement error (usually with the error term) to see if they can plausibly account for the effect. The separation of measurement error and sampling error is not important here. $\endgroup$
    – David Lane
    Commented Feb 17, 2017 at 0:17
  • 1
    $\begingroup$ @PeterFlom does not say it, but a reader might mistakenly infer from the first paragraph that non-experimental studies have an external-validity advantage. See Aronow and Samii (2015 AJPS) arguing that: "Causal effects estimated via multiple regression differentially weight each unit's contribution. The 'effective sample' that regression uses to generate the estimate may bear little resemblance to the population of interest," and therefore, "There is no general external validity basis for preferring multiple regression on representative samples over quasi-experimental or experimental methods." $\endgroup$
    – Dr. Beeblebrox
    Commented Jun 26, 2017 at 15:35
2
$\begingroup$

It is widely believed that the answer to your question is "yes".

Because of your second argument "Randomization does not guarantee that the two groups are balanced on all important factors at baseline" I am not convinced either.

In more detail, I considered several quantitative models to validate the often-heard claim that "[Randomization] only makes it unlikely there is severe unbalance".

However, in general, the last sentence is wrong. You may find the results in "Randomization does not help much" (see http://xxx.tau.ac.il/abs/1311.4390).

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ I am not sure whether this answer adds value to @Peter Flom's answer. In an infinitely large sample, randomization in fact guarantees the independence of the treatment of any other confounding factor. Of course, in any finite sample, it might well be the case that the treatment will be correlated with some other observed or unobserved factor. See Imai et al., "Misunderstandings between experimentalists and observationalists about causal inference." Journal of the royal statistical society: series A, 171.2, especially section 5, for a nice discussion of this problem and what to do about it. $\endgroup$
    – Julian Schuessler
    Commented May 5, 2014 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.