2
$\begingroup$

Imagine we have a categorical variable, A—for example, whether somebody owns a dog or not—and a quantitative variable, B—for example, how many days a person is sick in a given year. Assume that our measurements are valid and correct and the assumptions for a t-test are met. Imagine we have two studies:

  • In the first study, we send out a survey to a large random sample of people asking about both A and B, and then use a T-test to measure whether there is a statistically significant difference between dog owners and non-dog owners in how often they get sick.

  • In the second study, we have a representative group of volunteers. We randomly assign half of them to have a dog in their home and half not to for the next year, and then we measure how many days they are sick. We then use a T-test to measure whether there is a statistically significant difference between dog owners and non-dog owners in how often they get sick.

In the first study, there are a lot of reasons we might get a statistically significant difference:

  1. It could be random chance; we'd expect that to happen once in a while.
  2. It could be that A causes B, which is what we're looking for.
  3. It could also be that B causes A.
  4. It could be that some other variable, C, causes A and B.
  5. (There might also be other options.)

Is it correct to say that an experiment showing a statistically significant difference has the same potential explanations as an observational study showing such a statistically significant difference, except for B causing A and C causing A and B? If not, what are the other options and why do they exist for an observational, but not experimental, result?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Yes, it is correct to say that with an experiment, you have ruled out the reverse causal explanation.  And, unless you have other experimental designs that rule out the other plausible alternative explanations, then both the experimental and observational study will have the same plausible alternatives (e.g., same threats to validity).  The arguable benefit for the experimental design is that you have ruled out the reverse causal relationship, a.k.a., the temporal condition.  This brings us one step closer to Hume’s criteria for causality:
(1) A has an observable effect on B
(2) A occurs before B
(3) No other explanations exist.

You can argue (1) for both an observational and experimental study.  You have guaranteed (2) for an experimental study (though it may be true for an observational study, too, but it may be related to a confounding variable).  But, as you suggest in your query, (3) remains unanswered for both designs.

Happy to elaborate further, though a great reference for this would be Campbell & Stanley’s (1963) seminal work on experimental and quasi-experimental designs.  (Note:  they address the issue of randomization vs. representation in this work, though I chose not to elaborate here.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ But isn't it inherent in an experiment that a correlation couldn't result from a variable C causing A and B, because in an experiment we completely control A? For instance, if we observe whether people who eat kale are less likely to contract cancer, it might be that people who are health-conscious are eating kale, and are also doing other things that make them less likely to contract cancer. If we had an experiment where we randomly assigned some people to eat kale and others not to, we'd know that wasn't happening. Aren't we, then, inherently controlling for part of (3) in an experiment? $\endgroup$
    – Jean Luc Picard
    Commented Mar 22, 2018 at 18:18
  • $\begingroup$ Sorry, I think I may have phrased my parenthetical comment about confounding in 2nd ¶ a little weirdly. I meant to suggest that confounding would be an issue with observational designs, not experimental. Regarding the second query, the hope is that randomization is accounting for this, but the nature of randomization is such that we could randomly end up with very disparate groups (which would introduce confounding back into the picture). (Then, it becomes an issue of semantics: ¿did the randomization cause the result or the confounding variable?) $\endgroup$
    – Gregg H
    Commented Mar 22, 2018 at 18:35
0
$\begingroup$

I'm sure you'll get great answers, but will give you a different perspective than what's usually considered on this subject. I claim that it's not just the question of experimental design. It's also the phenomenon itself, its stability.

Consider this: we observe that more Sun in the skies is correlated with warmer climate. For instance, summer time there's more Sun, and it's warmer. In South there's more Sun, and it's warmer etc. You can't really test this directly experimentally. Yet, I doubt you'll find many people contesting this notion these days.

In this case we have a very stable phenomenon, and the totality of our knowledge of astronomy and physics would rule out any other explanation. Thus my claim is that it's not just about the design of the experiment, it's also other factors such as our prior knowledge and the nature of the phenomenon under study

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is a great addition. I will just add one comment that may make this argument weaker for the social sciences: temporal persistence. The notion of stability draws on this aspect of empiricism. And it can be reasonably argued that it makes sense for the physical sciences (e.g., the color blue remains blue and doesn't become green on a given date in the near future...the philosophical "grue"). But, persistence is not a given in the social sciences (e.g., what causes racism 50 years ago may be very different from what causes racism today...or not). $\endgroup$
    – Gregg H
    Commented Mar 22, 2018 at 18:41
  • $\begingroup$ @GreggH, your comment only strengthens my argument. In social sciences the issues due to observational nature of the study are more severe, precisely due to the inherent instability of relationships between variables. You can go through graduate physics program without ever encountering the concept of "observational study" or "endogeneity", while in econ programs this will be intorduced in the first year econometrics course. I also wrote here once that in the best physics experiments you don't need statistics, the setup of the experiment takes care of what we fight with in social sciences $\endgroup$
    – Aksakal
    Commented Mar 22, 2018 at 19:08
  • $\begingroup$ I agree..."weaker" wasn't the right word choice...this issue complicates social science research because of the lack of stability in many systems/contexts being studied...and thus observational studies provide more questionable evidence (in most cases) $\endgroup$
    – Gregg H
    Commented Mar 22, 2018 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.