What is the impact of doubling a sample size on a p-value Assuming that their is an underlying relationship between two variables in an OLS regression [null hypothesis testing], then what is the impact on the p-value of doubling the sample size? (assuming that the initial sample is representative of the population, and the subsequent sample is also representative).
Obviously i'm aware that so long as there is an underlying relationship, then increasing the sample size should reduce the p-value, but I'm interested in understanding further the nature of the relationship between p and n.
 A: For the T test we have rules like "Doubling the sample size increases the test statistic by $\sqrt{2}$ ". This might make you think that there is a simple relationship between sample size and p-value.
In fact the relationship between sample size and p-value depends on the relationship between sample size and the test statistic, and the relationship between test statistic and p-value. Those relationships will be different for every test.
For the most simple case, the one sided Z test, we can see what this relationship is. Suppose a random variable $X$ has mean $\mu$ and variance $\sigma^2$. Uppose that we are testing if the mean of $X$ is significantly different from $\nu$. The test statistic $Z$ is $\frac{(\bar{x}-\nu)\sqrt{n}}{\sigma}$.
The p value is equal to one minus the CDF of the $Z$ statistic (this assumes that the difference between means is positive, a similar argument works if the difference is negative).
For the normal distribution the CDF is $\Phi(t)=0.5+0.5\cdot erf(\frac{x-\mu_t}{\sigma_t \sqrt{2}})$. Where erf(x) is the error function.
Under the null hypothesis of equal means the $Z$ statistic has a mean $0$ and variance $1$. The actual distribution of $Z$ has a mean of $\frac{(\bar{x}-\nu)\sqrt{n}}{\sigma}$ and variance $1$.
The effect size of the difference between the means is $\frac{(\bar{x}-\nu)}{\sigma}$. Call the effect size $b$, then the expected value of $Z$ is $b\sqrt{n}$.
For $Z$ the CDF is $\Phi(z)=0.5+0.5\cdot erf(\frac{z}{\sqrt{2}})$. Where erf(x) is the error function.
Of course the $Z$ statistic is a random variable, here we'll just look at the relationship between sample size and p-value for the expected value of $Z$.
It follows that the CDF of the $Z$ statistic is $\Phi(z)=0.5+0.5\cdot erf(\frac{b\sqrt{n}}{\sqrt{2}})$
This is the relationship between the p value and sample size
$p=0.5-0.5\cdot erf(\frac{b\sqrt{n}}{\sqrt{2}})$
The relationship varies according to the value of $n$. For very large $n$ we can use a series expansion to see the limiting behavior. According to wolfram alpha that is:
$\lim_{n \to \infty}p = e^{-0.5b^2n} \left(\frac{1}{eb\sqrt{n}}+O\left(\frac{1}{(b\sqrt{n})^2} \right) \right)$
That is quite a quick decay towards 0. There is a big dependence on the effect size, of course if the difference between means is greater then the p value will shrink more quickly as your sampling improves.
Again, remember that this is only for the Z and T test, it doesn't apply to other tests.
A: Let us first investigate the effect on the t-value. We can then immediately infer the effect on p-value.
This is perhaps best illustrated by a well-chosen simulation example which illustrates the most salient features. Since we're looking at $H_0$ being false (and we're essentially considering properties related to power) it makes sense to focus on a one-tailed test (in the "correct" direction) since looking at the wrong tail won't see much action and won't tell us much of interest.
So here we have a situation (at n=100) where the effect is large enough that the statistic is sometimes significant. We then add to that first sample a second drawing from the same continuous distribution of x-values (here uniform but it's not critical to the observed effect) of the same size as the first, leading to a doubling of the sample size, but entirely including the first sample.

What we observe is not that the p-value goes down, only that it tends to go down (more points lie above the diagonal line than below it); we can see that the variation in t-values reduces, so there are fewer in the region of 0. Many p-values go up. Quite a number of samples that were insignificant became significant when we added more data, but some that were significant became insignificant.
[Here we're looking at the t-statistic for the slope coefficient in a simple regression, though qualitatively the issues are similar more broadly.]
A plot of p-values instead of t-values conveys essentially the same information. Indeed if you put tick-markings at the right intervals on the axes above, you could label them with p-values instead ... but the top (and right) would show low p-values and the bottom (/left) would be labelled with larger p-values. [Actually plotting the p-values just squashes everything up into the corner, and it's less clear what's going on.]
A: In general, when the respective null is false, expect decay of the p-values as in the figure below, where I report average p-values from little simulation study for multiples of samples of size n=25 ranging from bb*n=25to bb*n=29*25 for a simple linear regression coefficient equal to 0.1 and error standard deviation of $\sigma_u=0.5$.
Since the p-values are bounded from below by zero, the decay must ultimately flatten out.
The 90% confidence interval (shaded blue area) indicates that, moreover, the variability of the p-values also decreases with sample size.

Evidently, when either $\sigma_u$ is smaller or $n$ larger, the p-values will be close to zero faster when increasing bb, so that the appearance of the plot will be flatter.
Code:
    reps <- 5000
    B <- seq(1,30,by=2)
    n <- 25
    
    sigma.u <- .5
    pvalues <- matrix(NA,reps,length(B))
    for (bb in 1:length(B)){
         for (i in 1:reps){
              x <- rnorm(B[bb]*n)
              y <- .1*x + rnorm(B[bb]*n,sd=sigma.u)
              pvalues[i,bb] <- summary(lm(y~x))$coefficients[2,4]     
         }
    }
    plot(B,colMeans(pvalues),type="l", lwd=2, col="purple",  
            ylim=c(0,.9))
    ConfidenceInterval <- apply(pvalues, 2, quantile, 
                                 probs = c(.1,.9))
    x.ax <- c(B,rev(B))
    y.ax <- c(ConfidenceInterval[1,], 
                 rev(ConfidenceInterval[2,]))
    polygon(x.ax,y.ax, col=alpha("blue", alpha = .2), border=NA)

