Current examples where randomized studies yield different results than observational studies

Question

I have been reading the following paper by G. Smith (2002). I found it rather surprising. Some excerpts:

Promising epidemiological and laboratory findings led to a paper published in 1981 in Nature entitled “Can dietary beta-carotene materially reduce human cancer rates?”5 Cancers related to smoking seemed particularly tractable, and by 1990 the answer for lung cancer was a clear yes: Walter Willett, reviewing the observational epidemiological data, concluded that “Available data thus strongly support the hypothesis that dietary carotenoids reduce the risk of lung cancer.”6 Four years later a large scale randomised controlled trial showed an 18% increase (3% to 36%) in lung cancer in those taking β carotene.7 Vitamin E and coronary heart disease provided another example of observational studies and randomised controlled trials failing to reach the same conclusion.w3

“Eating fruit halves the risk of an early death” the Independent claimedw4 in an excited response to a study showing a strong inverse association between blood vitamin C levels and mortality due to coronary heart disease.8 A subsequent randomised controlled trial of a vitamin supplement that raised blood vitamin C levels by 15.7 μmol/l found five year mortality due to coronary heart disease unchanged (relative risk 1.06; 0.95 to 1.16),9 whereas the equivalent observational findings for this increase in blood vitamin C were coronary heart disease relative risks of 0.63 (0.49 to 0.84) in women and 0.72 (0.61 to 0.86) in men (see fig A on bmj.com). Again, the results from robust experiment and fallible observation are clearly non-compatible.

Are observational studies still unreliable? If not, what has been improved to get more reliable results? Are there any current examples of observational results which are not confirmed by randomized studies?

Answer 1

Interesting question. There is actually a wide literature regarding observational studies not confirmed by successive trials. I'll provide you, just as an example, the role of fatty acids in different conditions, such as diabetes: Fatty acids in diabetes NEJM.

Several efforts have been put to improve quality of evidence from observational studies, among which the use of "propensity score matching" are those leading the way at the moment (and I'd encourage you to study them if you're interested in medical research).

However, the point is not about if those results are "reliable", but rather how careful you should be when deciding to do something based on data from observational studies, since they are hugely influenced by external and population factors (such as the "self randomization").

In medical guidelines, evidence from only observational studies are usually expressed with the lower level of evidence possible (usually C, or B if there are multiple observational studies), suggesting that you should be careful when actually taking that decision before any RCT have been done to evaluate that result.

Also, if they where completely unreliable, no one would do them. The main point of observational studies is not to find straightfowardly the truth, but rather as hypothesis-generating studies, saying "Hey, we found this! Why don't you check if this is actually true?"

Hope this was helpful.