Relative size of p values at different sample sizes

How does the relative size of a p value change at different sample sizes? Like if you got $p=0.20$ at $n=45$ for a correlation and then at $n=120$ you got the same p value of 0.20, what would be the relative size of the p value for the second test, compared to the original p value when $n=45$?

• Please explain the sense in which you are modifying sample sizes. Are you trying to compare p-values for two independent experiments of different things or are you instead perhaps contemplating the possibility of augmenting a sample of size $45$ by collecting $120-45$ additional independent observations? – whuber Mar 24 '14 at 14:57
• Unfortunately I wasn't given any more information than that in the question – user42458 Mar 25 '14 at 1:30
• This is for some subject? – Glen_b -Reinstate Monica Mar 25 '14 at 2:19

Consider tossing a coin which you suspect may come up heads too often.

You perform an experiment, followed by a one tailed hypothesis test. In ten tosses you get 7 heads. Something at least as far from 50% could easily happen with a fair coin. Nothing unusual there.

If instead, you got 700 heads in 1000 tosses, a result at least as far from fair as that would be astonishing for a fair coin.

So 70% heads is not at all strange for a fair coin in the first case and very strange for a fair coin in the second case. The difference is sample size.

As the sample size increases, our uncertainty about where the population mean could be (the proportion of heads in our example) decreases. So larger samples are consistent with smaller ranges of possible population values - more values tend to become "ruled out" as samples get larger.

The more data we have, the more precisely we can pin down where the population mean could be... so a fixed value of the mean that is wrong will look less plausible as our sample sizes become large. That is, p-values tend to become smaller as sample size increases, unless $$H_0$$ is true.

• Thanks :) And how does that fit with getting the same p-value (not smaller) with a larger sample size? – user42458 Mar 25 '14 at 0:58
• Your question doesn't say the p-value is the same, it says you thought it would be the same. Is this a new question or were you just especially unclear about what you wanted? In any case, it can happen - if the larger sample is just enough closer to what you'd expect under the null to make it so. Imagine you had 8 heads on 25 tosses (32% heads), but 14 heads in 39 tosses (about 36% heads). The p-value for a test of $P(H)=0.5$ is almost the same. – Glen_b -Reinstate Monica Mar 25 '14 at 1:25
• Your edited question is now very confusing. I thought I understood what you were asking, now I have absolutely no clue what you're talking about. (Apparently what it looked like it was asking is not what it was asking.) – Glen_b -Reinstate Monica Mar 25 '14 at 1:35
• I don't know what is intended by the phrase 'relative p-value' there. – Glen_b -Reinstate Monica Mar 25 '14 at 2:20
• To the anonymous editor: 700 heads in 1000 tosses is far more than necessary to establish the point; it's already 12.65 standard deviations from the mean. It corresponds to a p-value of $1.7 \times 10^{-37}$. It's already an extreme example, so multiplying everything by 1000 does not actually help make that point better. Even 70 out of 100 would be more than enough. – Glen_b -Reinstate Monica Nov 12 '17 at 4:19

I agree with @Glen_b, just want to explain it from another point of view.

Let's put the example of the difference of means in two populations. Rejecting $H_{0}$ is equivalent to say that 0 is not in the confidence interval for the difference of means. This interval gets smaller with n (by definition), so it will become harder and harder for any point (in this case, the zero) to be in the interval as n grows. As rejection by confidence interval is mathematically equivalent to rejection by p-value, p-value will get smaller with n.

It will come the moment when you will get an interval like $[0.0001, 0.0010]$ that will indicate that the first population has indeed a bigger mean than the second population, but this difference is so little that you would not mind it. You will reject $H_0$, but this rejection wont mean anything in real life. That is the reason why p-values are not enough to describe a result. One must always give some measure of the SIZE of the observed difference.

The $p$ value for a of a that a given, nonzero is actually zero in the population will decrease with increasing sample size. This is because a larger sample that provides consistent evidence of that nonzero effect is providing more evidence against the null than a smaller sample. A smaller sample offers more opportunity for random sampling error to bias effect size estimates as @Glen_b's answer illustrates. Regression to the mean reduces sampling error as sample size increases; an effect size estimate based on a sample's central tendency improves with the sample's size following the central limit theorem. Therefore $p$ – i.e., probability of obtaining more samples of the same size and with effect sizes at least as strong as that of your sample if you draw them randomly from the same population, assuming the effect size in that population is actually zero – decreases as sample size increases and the sample's effect size remains unchanged. If effect size decreases or error variation increases as sample size increases, significance can remain the same.

Here's another simple example: the correlation between $x=\{1,2,3,4,5\}$ and $y=\{2,1,2,1,3\}$. Here, Pearson's $r=.378,t_{(3)}=.71,p=.53$. If I duplicate the data and test the correlation of $x=\{1,2,3,4,5,1,2,3,4,5\}$ and $y=\{2,1,2,1,3,2,1,2,1,3\}$, $r=.378$ still, but $t_{(3)}=1.15,p=.28$. It doesn't take many copies ($n$) to approach $\lim_{n\to\infty} p(n)=0$, shown here:

• When you reference the CLT I think you really are meaning to reference the law of large numbers. The CLT gives us approximate normality of the sampling distribution - which you don't really mention at all. – Dason Sep 27 '14 at 18:35