How to calculate the significance of a new predictor in a regression model if I change the sample size? I have a regression model with two predictors and I am interested in the significance of the second predictor (the more complex model). R tells me the significance level, but I have a large N and am wondering if it would still be significant with the exact same parameters but a smaller N. How can I test that?
 A: $\newcommand{\e}{\varepsilon}$It sounds like you want to look at the test statistic for a single coefficient as a function of the sample size. This won't be a formal test but it'll give you a sense of what will happen.
I'll explore this by replacing the sample size of $n$ with $n\zeta$ for a scaling parameter $\zeta>0$ and see what happens. I'll also assume that both $n$ and $\zeta n$ are "sufficiently large" since you say you have a large sample initially.
Suppose we have $y = X_n\beta + \e$ with $\e\sim\mathcal N(\mathbf 0, \sigma^2 I)$ and $X_n$ is $n\times p$ and full rank. Let $S_{j,n}^2 = (X_n^TX_n)^{-1}_{jj}$. Then for one coefficient we have
$$
  T_n := \frac{\hat\beta_j}{\hat\sigma S_{j,n}} \sim t_{n-p}
$$
where $\hat\sigma^2  =\frac 1{n-p}\|y - X\hat\beta\|^2$ (I give the full details of this in my answer here). I will assume the null is false, i.e. $\beta_j\neq 0$, so that $T_n$ is not converging in probability to $0$ (this is so I can reasonably look at $\frac{T_{\zeta n}}{T_{n}}$).
In the large sample case, $\frac{\hat\beta_j}{\hat\sigma} \stackrel{\text p}\to\frac{\beta_j}{\sigma}$ so even though we may be changing sample sizes, these quantities will be pretty similar either way (for $\zeta$ not too small). This means
$$
  \frac{T_{\zeta n}}{T_{n}} \approx \frac{S_{j,n}}{S_{j,\zeta n}}.
$$
If I'm adding new rows to $X$ in an iid fashion and I have a well-behaved distribution that the new rows are coming from, then $\frac 1n (X^T_n X_n)^{-1}$ will also converge to something. This means that
$$
\frac{T_{\zeta n}}{T_n} \approx \sqrt{
\frac{\zeta n \cdot \frac 1{\zeta n} (X_{\zeta n}^TX_{\zeta n})^{-1}_{jj}}{n \cdot \frac 1n (X_n^TX_n)^{-1}_{jj}}} \approx \sqrt \zeta
$$
so if all samples sizes concerned are sufficiently large, we'll see that the change in going from a sample size of $n$ to $\zeta n$ is to scale the original test statistic by approximately $\sqrt \zeta$. Furthermore, the reference distributions of $t_{n-p}$ and $t_{\zeta n-p}$ will both be effectively indistinguishable from $\mathcal N(0,1)$ in this large sample case, so we can get a rough sense of the significance of a new sample size $\zeta n$ just by scaling the original test statistics by $\sqrt\zeta$.
Here's a simulation checking this:
rm(list=ls())
get_test_stats <- function(n, p, betas, s2_err) {
  x <- cbind(1, matrix(rnorm(n*(p-1)), n, p-1))
  e <- rnorm(n, 0, sqrt(s2_err))
  y <- x %*% betas + e
  summary(lm(y~x-1))$coef[,"t value"]
}
set.seed(132)
n <- 100000
p <- 8
zeta <- .765
betas <- runif(p,-1,1)  # nulls all are false (almost surely)
s2_err <- .87

t_full <- get_test_stats(n, p, betas, s2_err)
t_reduced <- get_test_stats(n * zeta, p, betas, s2_err)
rbind(t_full, t_full * sqrt(zeta), t_reduced)

