2
$\begingroup$

I have come across (Ioannidis, 2005) which explains several reasons (mainly statistics-related, that's why I post this question here) to justify the claim that most published research is indeed false. Phys.org, Beware those scientific studies—most are wrong, researcher warns also treats the topic, mentioning a 2015 reproducibility test that puts the previous article into practice.

Has any author refuted this claim, or is the article right and most published research is actually false? In the latter case, shouldn't this be public knowledge?

$\endgroup$
10
  • 4
    $\begingroup$ You might want to edit this to ask more specifically if Ioannidis' statistical claims have merit. Other questions (eg, "why do people often back their claims with a link to an article?") fall less clearly within our scope. Note also, that as a matter of logic, it is possible no one has (yet) refuted the claims, but the article is nonetheless wrong. $\endgroup$ Commented Jun 11, 2019 at 15:34
  • $\begingroup$ You may be interested in similar threads, such as in stats.stackexchange.com/questions/200500/… and stats.stackexchange.com/questions/200745/… $\endgroup$
    – Sycorax
    Commented Jun 11, 2019 at 15:36
  • $\begingroup$ @gung Indeed it is, and paradoxically the chances of him being wrong increase if he is right! $\endgroup$
    – David
    Commented Jun 11, 2019 at 16:11
  • 3
    $\begingroup$ Please edit your question accordingly. $\endgroup$
    – Glen_b
    Commented Jun 12, 2019 at 1:05
  • 3
    $\begingroup$ This seems rather to be asking for a discussion, rather then asking a specific question. It seems oppinion based and broad in current form. $\endgroup$
    – Tim
    Commented Jun 12, 2019 at 8:40

1 Answer 1

5
$\begingroup$

To have a sensible discussion about this, we first need to clarify what we mean by "false". The title of the 2005 paper is obviously a short hand, intended to attract the curiosity of the reader.

If you read the paper, you will see what Ioannidis mainly points out is that applying the right (or at least the current standard) practice for scientific discovery, the rate of false positives ending up in the literature may be much higher than what most people intuitively believe (possibly > 50%) .

The basis of the argument is the following: assume people test a lot of things, and apply the p-value to see if there is an effect. Things that truly have an effect will return significant with a rate depending on the power of the study. Things that do not have an effect will return significant with the Type I error rate (= the significance level, usually 0.05). This is simply a mathematical fact.

Now, assume further (and this is the critical assumption) that we only publish things that come back as significant. The rest of the experiments is never reported. Whether this is the case is an empirical question, but it is probably a good approximation for how many labs and journals work.

If the latter assumption is true, we need to calculate which proportion of our significant results originate from a true effect. The opposite, i.e. the proportion that does not come from a true effect, is the false discovery rate (FDR). I have pasted a visualisation of this from my lecture notes below (sorry, only in German). If you do the maths, you will see that the FDR increases with the proportion of things in your test pool that truly have no effect, and increases when the power of the study is low (see also here).

In the 2005 paper, Ioannidis did a back-of-the-envelope calculation about FDRs in biomedical studies and arrived at 50%. Since then, many studies have looked at effect sizes and power to run similar analyses, with results pointing towards similar numbers. I would argue the fact that the current publishing system (due to publication bias towards significant results) produces FDRs way in the 2-digit area is established knowledge.

The solution, as pointed out by many, would be to make publication decisions independent of the effect size / p-value, for example by having an editorial process where decision about the acceptance of the paper only rests on introduction and methods, but this is of course not very popular with the journals, because they prefer to have studies that show effects.

Note that all this is independent of p-hacking and other ways of "statistical cheating", which might further distort results in the literature.

FDR

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.