# What is it called when an experimenter discards results that are too unexpected?

There is a type of scientific error where an experimenter gets a result significantly different from prior researchers, assumes they made a mistake, and redoes the experiment until they get a more expected value, which they publish. I vaguely remember hearing about this in a Feynman book or video, where he described how correcting the known value of constants took longer than it should have because of this effect.

What is this effect called, and what are some famous examples?

UPDATE

I reworded the question to clarify what I meant by an "unexpected" result. Helpful commenters identified the Feynman anecdote:

The other posts don't include a term for the error.

• As I recall, Feynman was referring to the centuries-long history of experiments to measure the speed of light, which consistently (grossly) underestimated the imprecision and inaccuracy in the results.
– whuber
Nov 5 '21 at 20:11
• Published estimates of the charge on the electron crept up over the years following Millikan's original experiment before plateauing at the currently accepted value - Feynman attributed this to each successive researcher's finding reasons to reject measurements too discrepant with previous estimates. Nov 6 '21 at 1:06
• Nov 6 '21 at 8:48
• "bayesian statistics" :D Nov 7 '21 at 1:28

Another answer has mentioned publication bias. However, that is not really what you were asking about, which is data dredging. A pertinent XKCD illustration is:

• I have seen this comic many times, but never realized the brilliance of investigating 20 colors ($\frac{1}{20} = 5\%$) Nov 6 '21 at 21:00
• That's a great cartoon, but I don't see how data dredging applies in this situation. Nov 6 '21 at 21:34
• @EllenSpertus: It applies because when you investigate 20 different colours at the 5% confidence level, you can expect on average 1 of them to show 'significant' deviation from the null hypothesis, simply by definition of confidence level. Data dredging means that instead of publishing all 20 experiments in one paper, you publish only the 1 that 'showed' significant deviation. That is also why another name for data dredging is p-hacking (typically with p = 1/20). Worse still if you intentionally do 20 experiments in order to find a 'significant' result to publish! Nov 7 '21 at 3:18
• @user21820 I see the problem: My original post was ambiguous. I will fix it to prevent the misinterpretation you drew from my imprecise wording. Nov 7 '21 at 3:51
• @EllenSpertus: Okay maybe you're asking about unconscious bias towards the expected outcome. In that case, the term "data dredging" may not be accurate, but it is not completely clear that "unconscious data dredging" is inaccurate. After all, the wikipedia article says "If they are not cautious, researchers using data mining techniques can be easily misled by these results.". Why say that if data dredging must be a deliberate act? The more general class of such errors is confirmation bias, but it includes far more than "redoing experiments". Nov 7 '21 at 8:42

One way of framing this is as publication bias, which occurs when the outcome of an experiment influences the decision of whether or not to publish the result. This is a well-known form of bias that infects academic research. I'm not familiar with any "famous" examples, but there are a few works in the medical field that decribe some non-famous examples in Wilmherst (2007).

Examples of publication bias is inherently difficult to detect, since the non-published parts of the example are non-published (and therefore difficult to detect). Generally speaking, publication bias is detected through statistical analysis of reported metrics in published works. Consequently, most of the known "examples" of publication bias in academic literature are inferences of publication bias coming solely from the published works.

– whuber
Nov 5 '21 at 22:58
• Not sure where I've referred to hypothesis testing here.
– Ben
Nov 5 '21 at 22:59
• (+1) Possibly relevant, is that several published studies have been retracted after someone else has tried to replicate results. (Sometimes this has been by people who got negative results before the publication and decided to re-do the work...) Nov 5 '21 at 23:32
• Isn't the principal form of publication bias the use of NHST to determine whether a result is publishable?
– whuber
Nov 6 '21 at 19:31
• @whuber: Maybe, maybe not; in any case, publication bias covers any situation where the decision to publish is affected by the results from the data, so it would also cover cases where one chooses not to publish based on any kind of inferential result.
– Ben
Nov 6 '21 at 21:25

Example: Based on a real experiment, names of people and the organization (along with inconsequential details) are omitted to protect the guilty.

In a study comparing two methods (1 and 2) of manufacture, $$n=100$$ items were tested until failure. (Larger observed values are better.) Summary statistics for results x1 and x2 of the samples were as below:

summary(x1); length(x1);  sd(x1)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.1099  2.8264  7.0881 10.0057 12.8520 46.9993
[1] 100
[1] 10.35345

summary(x2); length(x2);  sd(x2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.1196  3.2247  8.0975 11.1469 15.9245 56.6384
[1] 100
[1] 10.54756

boxplot(x1, x2, col="skyblue2", horizontal=T, notch=T)


Everyone's favorite was Method 2 (even though more costly), and it had the larger mean. But the overlapping notches in the boxes suggest no significant difference. Also, a pooled 2-sample t.test, which "must be OK" because of the large sample sizes, finds no significant difference. [This was before Welch t tests became popular.] Experimenters were hoping for evidence that Method 2 was significantly better.

t.test(x1,x2, var.eq=T)

Two Sample t-test

data:  x1 and x2
t = -0.77212, df = 198, p-value = 0.441
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-4.055797  1.773441
sample estimates:
mean of x mean of y
10.00571  11.14689


The consensus was that the "outliers were messing up the t test" and should be removed. [No one seemed to notice that the new outliers had appeared with the removal of the original ones.]

min(boxplot.stats(x1)$$out) [1] 28.41372 y1 = x1[x1 < 28.4] min(boxplot.stats(x2)$$out)
[1] 36.73661
y2 = x2[x2 < 36.7]

boxplot(y1,y2, col="skyblue2", horizontal=T, notch=T)


Now with the "cleaned-up data" y1 and y2, we have a t test significant (just) below 5% level. Great joy, the favorite won out.

t.test(y1, y2, var.eq=T)

Two Sample t-test

data:  y1 and y2
t = -1.9863, df = 186, p-value = 0.04847
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-4.37097702 -0.01493265
sample estimates:
mean of x mean of y
7.660631  9.853586


To 'confirm they got it right', a one-sided ("because we already know which method is best") two-sample Wilcoxon test finds a significant difference (at very nearly the 5% level, but "nonparametric test are not as powerful"):

wilcox.test(y1, y2, alt="less")\$p.val
[1] 0.05310917


Some years later when an economic crunch forced switching to cheaper Method 1, it became obvious that there was no practical difference between methods. In keeping with that revelation, I sampled the data for the current example in R as below:

set.seed(2021)
x1 = rexp(100, .1)
x2 = rexp(100, .1)


Note: You can Google and find an exact F-test to compare exponential samples, and it finds no difference, but nobody thought to use it at the time.

• That's an interesting example of this in action, but you didn't answer the question about what the name of this behavior is. Nov 6 '21 at 5:38
• Sorry, I thought that had already been answered by @Ben (+1). Maybe a symptom of P-hacking. Nov 6 '21 at 5:51
• Maybe it would be nice to explain what x1, x2 are exactly and whether high x is a desirable outcome because it took me a while before I understood what was going on. Nov 7 '21 at 16:33
• @AccidentalTaylorExpansion: In this version of the story xs are times to failure so bigger is better. Experimenters were hoping for a test showing Method2 was better, but should have been hoping for a test showing the truth (no significant difference). // I have edited my Answer to add a couple of sentences. Nov 7 '21 at 16:59