Relative size of p values at different sample sizes

Question

How does the relative size of a p value change at different sample sizes? Like if you got $p=0.20$ at $n=45$ for a correlation and then at $n=120$ you got the same p value of 0.20, what would be the relative size of the p value for the second test, compared to the original p value when $n=45$?

Answer 1

Consider tossing a coin which you suspect may come up heads too often.

You perform an experiment, followed by a one tailed hypothesis test. In ten tosses you get 7 heads. Something at least as far from 50% could easily happen with a fair coin. Nothing unusual there.

If instead, you got 700 heads in 1000 tosses, a result at least as far from fair as that would be astonishing for a fair coin.

So 70% heads is not at all strange for a fair coin in the first case and very strange for a fair coin in the second case. The difference is sample size.

As the sample size increases, our uncertainty about where the population mean could be (the proportion of heads in our example) decreases. So larger samples are consistent with smaller ranges of possible population values - more values tend to become "ruled out" as samples get larger.

The more data we have, the more precisely we can pin down where the population mean could be... so a fixed value of the mean that is wrong will look less plausible as our sample sizes become large. That is, p-values tend to become smaller as sample size increases, unless $H_0$ is true.

Answer 2

I agree with @Glen_b, just want to explain it from another point of view.

Let's put the example of the difference of means in two populations. Rejecting $H_{0}$ is equivalent to say that 0 is not in the confidence interval for the difference of means. This interval gets smaller with n (by definition), so it will become harder and harder for any point (in this case, the zero) to be in the interval as n grows. As rejection by confidence interval is mathematically equivalent to rejection by p-value, p-value will get smaller with n.

It will come the moment when you will get an interval like $[0.0001, 0.0010]$ that will indicate that the first population has indeed a bigger mean than the second population, but this difference is so little that you would not mind it. You will reject $H_0$, but this rejection wont mean anything in real life. That is the reason why p-values are not enough to describe a result. One must always give some measure of the SIZE of the observed difference.

Answer 3

The $p$ value for a significance-test of a null-hypothesis that a given, nonzero effect-size is actually zero in the population will decrease with increasing sample size. This is because a larger sample that provides consistent evidence of that nonzero effect is providing more evidence against the null than a smaller sample. A smaller sample offers more opportunity for random sampling error to bias effect size estimates as @Glen_b's answer illustrates. Regression to the mean reduces sampling error as sample size increases; an effect size estimate based on a sample's central tendency improves with the sample's size following the central limit theorem. Therefore $p$ – i.e., probability of obtaining more samples of the same size and with effect sizes at least as strong as that of your sample if you draw them randomly from the same population, assuming the effect size in that population is actually zero – decreases as sample size increases and the sample's effect size remains unchanged. If effect size decreases or error variation increases as sample size increases, significance can remain the same.

Here's another simple example: the correlation between $x=\{1,2,3,4,5\}$ and $y=\{2,1,2,1,3\}$. Here, Pearson's $r=.378,t_{(3)}=.71,p=.53$. If I duplicate the data and test the correlation of $x=\{1,2,3,4,5,1,2,3,4,5\}$ and $y=\{2,1,2,1,3,2,1,2,1,3\}$, $r=.378$ still, but $t_{(3)}=1.15,p=.28$. It doesn't take many copies ($n$) to approach $\lim_{n\to\infty} p(n)=0$, shown here: