What to conclude when you fail to find an association in an epidemiological study?

Question

Normally when somebody finds an association in an epidemiological study people are quick to point out that it doesn't prove causality, that there are problems of missing co-founders, that it is at best hypothesis generating and at worst spurious. This leads to people not putting much weight on associations found in epidemiological studies.

What if it goes the other way around? Say I already have a theory, maybe with some small earlier studies to back it up and even a good theoretical explanation for the effect. Then I do a big, well powered, epidemiological study and fail to find the association the theory predicts. How much weight can I put on the result now?

Intuitively it seems to me that the result would be quite damning, despite it being only a epidemiological study. But I have long since learned not to trust my intuition when it comes to statistics.

Do all the weaknesses of associations found in epidemiological studies also apply to when you fail to find an association?

Answer 1

Yes, the weaknesses of associations found in epidemiological studies also apply to a failure to find an association. You've already eliminated the first go-to problem, that of a study being underpowered, so at the moment we're just talking about bias. Two issues that may mean your study is failing to find a true association:

• Confounding. There are conditions where a confounding variable will drive a result toward the null. For positive effects, this is the confounding variable being negatively associated with both the exposure and the outcome. For negative effects, the reverse. This could easily drive, depending on the strength of the confounding, a real relationship downward enough that you cannot find it.
• Misclassification. Non-differential misclassification is when everyone in your study has an equal probability of being in the wrong category. This tends to drive estimates toward the null. Differential misclassification, where particular categories are more prone to being misclassified, can drive results toward or away from the null.

So no, the results of a single study should never be taken as definitive "proof", one way or the other, of a causal relationship.

Answer 2

I think it's worth distinguishing a few aspects of the problem:

• Precision: If you have a bigger sample size, you will typically be able to estimate parameters with greater precision whether you define this in a frequentist sense in terms of smaller confidence intervals or in a Bayesian sense in terms of smaller credibility intervals. Thus, if you conduct an epidemiological observational study with a bigger sample you will have greater precision in describing the magnitude of a parameter of interest. This is true whether the parameter is a simple correlation coefficient or a regression coefficient in a broader model with many other predictors. It's also true whether the parameter of interest is derived from an observational study or an experimental study. So for example, you might get a very accurate estimate of the correlation between eating chocolate and Body Mass Index. Associations, whether causal or not, are real and interesting.

• Generalisation: However, even if you know the value of a parameter in your particular sample, there are still issues of generalisation. In epidemiology there are plenty of issues related to generalising across time, culture, social groups, and so on. Often we have theories and empirical evidence to guide us in this process of generalising. For example, we may argue that is safe to generalise a chocolate-BMI association over reasonable periods of time, but that perhaps across nations it is more complex, perhaps because of different eating and exercise habits, etc.

• Causality versus association: However, you seem to be particular interested in causal inference. At a basic level, the absence of an association in an observational study does not prove the absence of causality, just as the presence of an association in an observational study does not prove causality. Even if observational studies showed no relationship between, for example, chocolate and BMI, this would not prevent experimental studies from showing that when kids were fed more chocolate, they put on weight. The association or lack of association in an observational study may be informative as to causal processes, but it is not definitive. You still need to think hard about the theorised underlying causal processes.

Answer 3

Well, in physics we are very used to what you are saying and, to me at least, it is in fact far more exciting when this happens that when some researcher confirms theory. In these cases you just report that: you've found no evidence of what the theory predicts.

After checking your results and the possibe flaws of your study, it is also good practice to elaborate (at least qualitatively) your own guesses about why theory fails in your case.