In 1971, Amos Tversky and Daniel Kahnemann published an influential paper: BELIEF IN THE LAW OF SMALL NUMBERS.
They start the paper describing an experiment:
"Suppose you have run an experiment on 20 subjects, and have obtained a significant result which confirms your theory (z = 2.23, p < .05, two-tailed). You now have cause to run an additional group of 10 subjects. What do you think the probability is that the results will be significant, by a one-tailed test, separately for this group?"
(How would you answer, dear reader?)
Two small groups gave answers, with 0.85 being the median answer.
"However, .48 happens to be a much more reasonable estimate than .85"
QUESTION: How could this be (re-)computed exactly?
Additional wish: Perhaps, you know a source in the literature discussing the experiment in detail?
Additional infos:
In a footnote, Tversky and Kahnemann give some hints on how to obtain such a reasonable estimate. Here is a brief excerpt:
The required estimate can be interpreted in several ways. ... In the special case of a test of a mean with known variance, one would compute the power of the test against the hypothesis that the population mean equals the mean of the first sample. Since the size of the second sample is half that of the first, the computed probability of obtaining z >= 1.645 is only .473.
With regard to a bayesian approach, they conclude: "Assuming a uniform prior, the desired posterior probability is .478."
Thanks in advance!
