Probability of significant result in a second sample given first was significant

Question

It's an example mentioned in Amos Tversky and Daniel Kahneman's Belief in the Law of Small Numbers:

Suppose you have run an experiment on 20 subjects, and have obtained a significant result which confirms your theory ($z = 2.23$, $p < .05$, two-tailed). You now have cause to run an additional group of 10 subjects. What do you think the probability is that the results will be significant, by a one-tailed test, separately for this group?

The explanation in the essay, however, is a little hard to understand. It says as following:

On the other hand, if you feel that the probability is around $.48$, you belong to a minority. Only 9 of our 84 respondents gave answers between $.40$ and $.60$. However, $.48$ happens to be a much more reasonable estimate than $.85$. The required estimate can be interpreted in several ways. One possible approach is to follow common research practice, where a value obtained in one study is taken to define a plausible alternative to the null hypothesis. The probability requested in the question can then be interpreted as the power of the second test (i.e., the probability of obtaining a significant result in the second sample) against the alternative hypothesis defined by the result of the first sample. In the special case of a test of a mean with known variance, one would compute the power of the test against the hypothesis that the population mean equals the mean of the first sample. Since the size of the second sample is half that of the first, the computed probability of obtaining $z > 1.645$ is only $.473$. A theoretically more justifiable approach is to interpret the requested probability within a Bayesian framework and compute it relative to some appropriately selected prior distribution. Assuming a uniform prior, the desired posterior probability is $.478$. Clearly, if the prior distribution favors the null bypothesis, as is often the case, the posterior probability will be even smaller.

I'm not sure how we could get $1.645$ and $0.473$. Thanks for some detailed explanation.

Answer 1

The number $1.645$ comes about because it is the critical value for a one-sided z-test with $\alpha=.05$.

> qnorm(.95)
# 1.644854

This admittedly could have been made a bit clearer in the text. Anyway, the probability $.473$ then comes from taking the area above this critical value of a normal distribution centered at $2.23/\sqrt{2}$ (and with standard deviation = 1).

> pnorm(1.645, mean=2.23/sqrt(2), lower.tail=FALSE)
# 0.4728324

$2.23$ was the value of the test statistic in the original study, and this would also be (by assumption) the expected value of the test statistic in the replication study if the replication study has the same sample size as the original study -- but it doesn't, it has half the sample size of the original study, so we divide the expected test statistic by $\sqrt{2}$ to account for that.