Imagine we have a categorical variable, A—for example, whether somebody owns a dog or not—and a quantitative variable, B—for example, how many days a person is sick in a given year. Assume that our measurements are valid and correct and the assumptions for a t-test are met. Imagine we have two studies:
In the first study, we send out a survey to a large random sample of people asking about both A and B, and then use a T-test to measure whether there is a statistically significant difference between dog owners and non-dog owners in how often they get sick.
In the second study, we have a representative group of volunteers. We randomly assign half of them to have a dog in their home and half not to for the next year, and then we measure how many days they are sick. We then use a T-test to measure whether there is a statistically significant difference between dog owners and non-dog owners in how often they get sick.
In the first study, there are a lot of reasons we might get a statistically significant difference:
- It could be random chance; we'd expect that to happen once in a while.
- It could be that A causes B, which is what we're looking for.
- It could also be that B causes A.
- It could be that some other variable, C, causes A and B.
- (There might also be other options.)
Is it correct to say that an experiment showing a statistically significant difference has the same potential explanations as an observational study showing such a statistically significant difference, except for B causing A and C causing A and B? If not, what are the other options and why do they exist for an observational, but not experimental, result?