In 1971, Amos Tversky and Daniel Kahnemann published an influential paper: BELIEF IN THE LAW OF SMALL NUMBERS.

They start the paper describing an experiment:

"Suppose you have run an experiment on 20 subjects, and have obtained a significant result which confirms your theory (z = 2.23, p < .05, two-tailed). You now have cause to run an additional group of 10 subjects. What do you think the probability is that the results will be significant, by a one-tailed test, separately for this group?"

(How would you answer, dear reader?)

Two small groups gave answers, with 0.85 being the median answer.

"However, .48 happens to be a much more reasonable estimate than .85"

QUESTION: How could this be (re-)computed exactly?

Additional wish: Perhaps, you know a source in the literature discussing the experiment in detail?

Additional infos:

In a footnote, Tversky and Kahnemann give some hints on how to obtain such a reasonable estimate. Here is a brief excerpt:

The required estimate can be interpreted in several ways. ... In the special case of a test of a mean with known variance, one would compute the power of the test against the hypothesis that the population mean equals the mean of the first sample. Since the size of the second sample is half that of the first, the computed probability of obtaining z >= 1.645 is only .473.

With regard to a bayesian approach, they conclude: "Assuming a uniform prior, the desired posterior probability is .478."

Thanks in advance!

  • $\begingroup$ What exactly are you asking to "recompute"? The 0.48 value or the data from the experiment? If it's the former, why would information about the experiment be relevant? If it's the latter, how could we possibly hope to recover the data from a single statistic like "0.85"? $\endgroup$ – whuber Feb 22 '18 at 15:47
  • $\begingroup$ Of interest is a plausible explanation of the values .473 and .478 given the (somewhat meager) information on the (fictitious) experiments with n=20 and n=10. The 0.85 is just the median "intuition" of the expert, a result of the survey. $\endgroup$ – Al Jannico Feb 22 '18 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.