# Replication Probabilities

Imagine observing a random variable over time, let's say N = 20 years.

I compute the mean of the random variable within each year and obtain N estimates of the within year mean.

For each N estimates I compute the p-value that the estimate is different than zero via a t-test.

The question: what assumptions are needed and how does one compute the probability of observing 10 significant estimates?

My initial guess is to assume independence between estimates and compute:

0.05^10 x (1-0.95)^10 = a very small number

This feels incorrect. What am I missing?

Thanks, Paul

• The factor $\binom{20}{10}=184756.$ Even so, your calculation is unlikely (correctly) to answer any relevant statistical question about the situation. Could you explain how you propose to use this number?
– whuber
Commented Jan 6, 2022 at 18:42
• Right sorry that was a typo. The calculation $\binom{20}{10} \times 0.05^{10} \times (1-0.95)^{10} =$ very small number feels likely incorrect for the purpose of the question I'm trying to answer (as you say). The question is want to use the number for is the following. Imagine observing a daily stock return. Each year in your sample you compute the mean and a p-value from a t-test. Your sample is 20 years long. What is the probability of observing 10 estimates which are significant at the 5% level ? I hope that makes sense. Commented Jan 6, 2022 at 20:08
• It makes sense, but (1) it's an unlikely null hypothesis to be testing (if you observed exactly 10 out of 20 low p-values, and formulated your hypothesis based on that observation, then this calculation is illegitimate); (2) the likelihood of strong serial correlation over time renders the assumption of independence untenable; and (3) replacing the original data and some underlying meaningful question with a question about p-values loses so much information that one has to ask why such a procedure would be undertaken in the first place.
– whuber
Commented Jan 6, 2022 at 20:48
• re (1). Start with assuming this were a repeated experiment and the null was a mean zero normal. Under this null we should expect statistically significant estimates by chance 5% of the time. So by chance we expect 1 year to be significantly different than zero out of 20. Commented Jan 6, 2022 at 21:11
• Guess I'm thinking about a joint null that all years are zero. It sounds like an F-test in which case I could account for the serial dependence which might take care of your second point. Commented Jan 6, 2022 at 21:11