7
$\begingroup$

There are 100 clinical trials. I have 95% confidence and report five of them that are successful. Then, I claim my drug works.

What is this called, when I only publish good results and rely purely on random error? It's publication bias, but there must be a general term.

https://www.bmj.com/content/352/bmj.i637

The proportion of clinical trials published within 24 months of study completion ranged from 10.8% (4/37) to 40.3% (31/77) across academic medical centers, whereas results reporting on ClinicalTrials.gov ranged from 1.6% (2/122) to 40.7% (72/177)

Do they seriously have 95% confidence and report 5% of results?

$\endgroup$
0

5 Answers 5

11
$\begingroup$

$p$-value hacking

I’ve learned that the headline-grabbing cases of misconduct and fraud are mere distractions. The state of our science is strong, but it’s plagued by a universal problem: Science is hard — really f**ing hard. If we’re going to rely on science as a means for reaching the truth — and it’s still the best tool we have — it’s important that we understand and respect just how difficult it is to get a rigorous result. I could pontificate about all the reasons why science is arduous, but instead I’m going to let you experience one of them for yourself. Welcome to the wild world of $p$-hacking.

From an introductory paragraph at "Science isn't broken," a feature at Fivethirtyeight.com (Christie Aschwanden, Aug. 19, 2015).

The article describes how you can achieve publishable results (and reject a null hypothesis) even though the results are not reproducible.

The $p$-value is that "due to random chance" footnote that you are looking for. By hacking it, you can get your results published.

$\endgroup$
1
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – tchrist
    Dec 31, 2020 at 0:54
9
$\begingroup$

Cherry picking, suppressing evidence, or the fallacy of incomplete evidence ... you can check Wikipedia:

Cherry picking, suppressing evidence, or the fallacy of incomplete evidence is the act of pointing to individual cases or data that seem to confirm a particular position while ignoring a significant portion of related and similar cases or data that may contradict that position.

$\endgroup$
3
  • $\begingroup$ I like the cherry picking idiom. It is entirely appropriate to the circumstances and I commend it to the questioner as a snappy and easily understood analogy. $\endgroup$
    – Anton
    Dec 30, 2020 at 15:15
  • $\begingroup$ ... But since 'cherry picking' was given in the earlier incarnation of this question, not in line with ELU aims. $\endgroup$
    – Edwin Ashworth
    Dec 30, 2020 at 15:25
  • 1
    $\begingroup$ There are many, many instances of "cherry-picking" being given as an answer. See: english.stackexchange.com/search?q=cherry+picking Unfortunately, the OP does not accept that idea. I think the question was badly worded, and possibly does not belong here. $\endgroup$
    – Cascabel
    Dec 30, 2020 at 17:55
5
$\begingroup$

Although the technical details may deter folk from answering, they are secondary to your main question about English terminology.

Nevertheless, let's get them out of the way. You have 95% confidence in the results; I take this to mean you knew that 5 in a hundred fail. The chance of randomly picking 1 success from a batch of 100 is 95/100, leaving 99 to choose from. A second success of 1 from 99 has chance 94/99, and so forth. It works out that the chance of randomly picking 5 successes from 100 is about 77%.

But you have presented 5 successes to the world as evidence that the drug works with 100% confidence rather the 77% confidence that you should have inferred. This is at least statistical misrepresentation.

Even if you chose your five examples at random, you have chosen to present a circumstance that happens only 77% of the time in such trials as one that always happens. This is unwarranted inference.

Or you may have chosen your five examples knowing them to be successes. This is selective bias and is statistical falsification.

$\endgroup$
3
  • $\begingroup$ Are these recognised terms or free combinations? They are represented (italicisation) as being at least strong collocations, but is this misrepresentation? References? $\endgroup$
    – Edwin Ashworth
    Dec 30, 2020 at 17:37
  • 1
    $\begingroup$ @EdwinAshworth In view of the controversy over this question I think it best to now leave the phrases as "an exercise for the student" rather than plod on with search that anyone may easily make themselves.. $\endgroup$
    – Anton
    Dec 30, 2020 at 17:50
  • $\begingroup$ I read OPs setup differently. I read it as as only 5 out of 100 were successful with 95% confidence. $\endgroup$
    – Jim
    Dec 30, 2020 at 20:59
2
$\begingroup$

If the trials are being tested with a variety of statistical methods, and only the "successful" ones publicized, then the primary term for this would be data dredging. From Wikipedia:

Data dredging (also data fishing, data snooping, data butchery, and p-hacking) is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. This is done by performing many statistical tests on the data and only reporting those that come back with significant results.

On the other hand, if only a single statistical test is being applied, and the failed trials literally discarded, then "cherry picking" (from another answer) would be the better term.

$\endgroup$
1
$\begingroup$

There's the term publication bias, but that's more about studies done by different researchers where only the researchers who get "good" results publish them. A similar term is "file drawer effect".

The term p-hacking doesn't quite apply, as p-hacking refers to cherry-picking a metric for a particular data set, not cherry-picking the data set. For instance, if you're testing for ESP, and you have someone guess playing cards, you can look at how many times they get the exact card, how often they get the right card value, how often they get the right suit, how often they get the right color, etc. If you keep looking at different ways of measuring "success" until you get one with p<0.05, that's p-hacking.

$\endgroup$
2
  • $\begingroup$ I think your distinction is artificial, and p-hacking does apply. p-hacking / data dredging / selective reporting is defined as "the misuse of data analysis to find patterns in data that can be presented as statistically significant". It doesn't require that people use any metric. Sure, we would expect that people who want to selectively report would dress up some excuse (not necessarily a well-defined metric) to avoid blatantly committing malpractice, but if there are enough extraneous variables ("include only tests performed when the sun was shining in Philadelphia")... $\endgroup$
    – smci
    Jan 1, 2021 at 12:50
  • $\begingroup$ ...they could construct an arbitrary and spurious narrative ("metric") to support any arbitrary choice of subset. Depending on the discipline (it's not always pharmaceutical trials), I'm not aware that researchers are even required to explicitly state any metric, let alone a non-spurious one, they just silently do the cherrypicking and only mention the subset of data that shows the result they want. $\endgroup$
    – smci
    Jan 1, 2021 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy