10
$\begingroup$

There is a type of scientific error where an experimenter gets a result significantly different from prior researchers, assumes they made a mistake, and redoes the experiment until they get a more expected value, which they publish. I vaguely remember hearing about this in a Feynman book or video, where he described how correcting the known value of constants took longer than it should have because of this effect.

What is this effect called, and what are some famous examples?

UPDATE

I reworded the question to clarify what I meant by an "unexpected" result. Helpful commenters identified the Feynman anecdote:

The other posts don't include a term for the error.

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ As I recall, Feynman was referring to the centuries-long history of experiments to measure the speed of light, which consistently (grossly) underestimated the imprecision and inaccuracy in the results. $\endgroup$
    – whuber ♦
    Nov 5, 2021 at 20:11
  • 7
    $\begingroup$ Published estimates of the charge on the electron crept up over the years following Millikan's original experiment before plateauing at the currently accepted value - Feynman attributed this to each successive researcher's finding reasons to reject measurements too discrepant with previous estimates. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Nov 6, 2021 at 1:06
  • 2
    $\begingroup$ Related backgound: Is Feynman's claim about the history of measurements of the charge of the electron after Millikan accurate? $\endgroup$
    – Simon
    Nov 6, 2021 at 8:48
  • 2
    $\begingroup$ "bayesian statistics" :D $\endgroup$
    – Jacob Socolar
    Nov 7, 2021 at 1:28

3 Answers 3

Reset to default
17
$\begingroup$

Another answer has mentioned publication bias. However, that is not really what you were asking about, which is data dredging. A pertinent XKCD illustration is:

https://xkcd.com/882

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ I have seen this comic many times, but never realized the brilliance of investigating 20 colors ($\frac{1}{20} = 5\%$) $\endgroup$
    – Steven Gubkin
    Nov 6, 2021 at 21:00
  • $\begingroup$ That's a great cartoon, but I don't see how data dredging applies in this situation. $\endgroup$
    – Ellen Spertus
    Nov 6, 2021 at 21:34
  • 6
    $\begingroup$ @EllenSpertus: It applies because when you investigate 20 different colours at the 5% confidence level, you can expect on average 1 of them to show 'significant' deviation from the null hypothesis, simply by definition of confidence level. Data dredging means that instead of publishing all 20 experiments in one paper, you publish only the 1 that 'showed' significant deviation. That is also why another name for data dredging is p-hacking (typically with p = 1/20). Worse still if you intentionally do 20 experiments in order to find a 'significant' result to publish! $\endgroup$
    – user21820
    Nov 7, 2021 at 3:18
  • $\begingroup$ @user21820 I see the problem: My original post was ambiguous. I will fix it to prevent the misinterpretation you drew from my imprecise wording. $\endgroup$
    – Ellen Spertus
    Nov 7, 2021 at 3:51
  • 1
    $\begingroup$ @EllenSpertus: Okay maybe you're asking about unconscious bias towards the expected outcome. In that case, the term "data dredging" may not be accurate, but it is not completely clear that "unconscious data dredging" is inaccurate. After all, the wikipedia article says "If they are not cautious, researchers using data mining techniques can be easily misled by these results.". Why say that if data dredging must be a deliberate act? The more general class of such errors is confirmation bias, but it includes far more than "redoing experiments". $\endgroup$
    – user21820
    Nov 7, 2021 at 8:42
11
$\begingroup$

One way of framing this is as publication bias, which occurs when the outcome of an experiment influences the decision of whether or not to publish the result. This is a well-known form of bias that infects academic research. I'm not familiar with any "famous" examples, but there are a few works in the medical field that decribe some non-famous examples in Wilmherst (2007).

Examples of publication bias is inherently difficult to detect, since the non-published parts of the example are non-published (and therefore difficult to detect). Generally speaking, publication bias is detected through statistical analysis of reported metrics in published works. Consequently, most of the known "examples" of publication bias in academic literature are inferences of publication bias coming solely from the published works.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The question actually asks about estimation rather than hypothesis testing. $\endgroup$
    – whuber ♦
    Nov 5, 2021 at 22:58
  • 1
    $\begingroup$ Not sure where I've referred to hypothesis testing here. $\endgroup$
    – Ben
    Nov 5, 2021 at 22:59
  • 4
    $\begingroup$ (+1) Possibly relevant, is that several published studies have been retracted after someone else has tried to replicate results. (Sometimes this has been by people who got negative results before the publication and decided to re-do the work...) $\endgroup$
    – BruceET
    Nov 5, 2021 at 23:32
  • $\begingroup$ Isn't the principal form of publication bias the use of NHST to determine whether a result is publishable? $\endgroup$
    – whuber ♦
    Nov 6, 2021 at 19:31
  • $\begingroup$ @whuber: Maybe, maybe not; in any case, publication bias covers any situation where the decision to publish is affected by the results from the data, so it would also cover cases where one chooses not to publish based on any kind of inferential result. $\endgroup$
    – Ben
    Nov 6, 2021 at 21:25
3
$\begingroup$

Example: Based on a real experiment, names of people and the organization (along with inconsequential details) are omitted to protect the guilty.

In a study comparing two methods (1 and 2) of manufacture, $n=100$ items were tested until failure. (Larger observed values are better.) Summary statistics for results x1 and x2 of the samples were as below:

summary(x1); length(x1);  sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.1099  2.8264  7.0881 10.0057 12.8520 46.9993 
[1] 100
[1] 10.35345

summary(x2); length(x2);  sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.1196  3.2247  8.0975 11.1469 15.9245 56.6384 
[1] 100
[1] 10.54756

boxplot(x1, x2, col="skyblue2", horizontal=T, notch=T)

enter image description here

Everyone's favorite was Method 2 (even though more costly), and it had the larger mean. But the overlapping notches in the boxes suggest no significant difference. Also, a pooled 2-sample t.test, which "must be OK" because of the large sample sizes, finds no significant difference. [This was before Welch t tests became popular.] Experimenters were hoping for evidence that Method 2 was significantly better.

t.test(x1,x2, var.eq=T)

          Two Sample t-test

data:  x1 and x2
t = -0.77212, df = 198, p-value = 0.441
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -4.055797  1.773441
sample estimates:
mean of x mean of y 
 10.00571  11.14689

The consensus was that the "outliers were messing up the t test" and should be removed. [No one seemed to notice that the new outliers had appeared with the removal of the original ones.]

min(boxplot.stats(x1)$out)
[1] 28.41372
y1 = x1[x1 < 28.4]
min(boxplot.stats(x2)$out)
[1] 36.73661
y2 = x2[x2 < 36.7]

boxplot(y1,y2, col="skyblue2", horizontal=T, notch=T)

enter image description here

Now with the "cleaned-up data" y1 and y2, we have a t test significant (just) below 5% level. Great joy, the favorite won out.

t.test(y1, y2, var.eq=T)

        Two Sample t-test

data:  y1 and y2
t = -1.9863, df = 186, p-value = 0.04847
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
  -4.37097702 -0.01493265
sample estimates:
mean of x mean of y 
 7.660631  9.853586

To 'confirm they got it right', a one-sided ("because we already know which method is best") two-sample Wilcoxon test finds a significant difference (at very nearly the 5% level, but "nonparametric test are not as powerful"):

wilcox.test(y1, y2, alt="less")$p.val
[1] 0.05310917

Some years later when an economic crunch forced switching to cheaper Method 1, it became obvious that there was no practical difference between methods. In keeping with that revelation, I sampled the data for the current example in R as below:

set.seed(2021)
x1 = rexp(100, .1)
x2 = rexp(100, .1)

Note: You can Google and find an exact F-test to compare exponential samples, and it finds no difference, but nobody thought to use it at the time.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ That's an interesting example of this in action, but you didn't answer the question about what the name of this behavior is. $\endgroup$
    – nick012000
    Nov 6, 2021 at 5:38
  • 4
    $\begingroup$ Sorry, I thought that had already been answered by @Ben (+1). Maybe a symptom of P-hacking. $\endgroup$
    – BruceET
    Nov 6, 2021 at 5:51
  • $\begingroup$ Maybe it would be nice to explain what x1, x2 are exactly and whether high x is a desirable outcome because it took me a while before I understood what was going on. $\endgroup$
    – AccidentalTaylorExpansion
    Nov 7, 2021 at 16:33
  • 1
    $\begingroup$ @AccidentalTaylorExpansion: In this version of the story xs are times to failure so bigger is better. Experimenters were hoping for a test showing Method2 was better, but should have been hoping for a test showing the truth (no significant difference). // I have edited my Answer to add a couple of sentences. $\endgroup$
    – BruceET
    Nov 7, 2021 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.