Z-score and standard error in linear regression

Question

I am reading Elements of statistical learning, and in the chapter on linear regression, I cannot understand the following:

We have estimated the regression parameters $\beta_1, ..., \beta_p$ from $N$ data points. It has been established that $\hat{\beta} \sim N(\beta, (X^T X)^{-1}\sigma^2)$, where $\beta$ is the true value of $\hat{\beta}$ and $X$ the data matrix. $\sigma^2$ is the variance of $Y$ around its mean, i.e. $Y = X\beta + \epsilon$, $\epsilon \sim N(0, \sigma^2)$.

Now, the authors calculate $z_j = \frac{\hat{\beta}_j}{\hat{\sigma}\sqrt{(X^TX)^{-1}_{jj}}}$. Also, the variance estimate is $\hat{\sigma}^2 = \frac{1}{N-p-1}RSS$.

Now, this is more or less clear, however I should also get the same $z_j$ by dividing $\hat{\beta}_j$ by its standard error. In other words, it should hold that $SE(\hat{\beta}_j) = \hat{\sigma}\sqrt{(X^TX)^{-1}_{jj}}$. But instead, I get the following: $SE(\hat{\beta}_j) = \frac{\sqrt{(X^TX)^{-1}_{jj}}\sigma}{\sqrt{N}} = \frac{\sqrt{(X^TX)^{-1}_{jj}(N-p-1)}\hat{\sigma}}{\sqrt{N}}$

where I used $(N-p-1)\hat{\sigma} \sim \sigma^2\mathcal{X}^2_{N-p-1}$

So something is off with the $\sqrt{N-p-1}$-factor, and I am afraid that I fundamentally misunderstood something here.

EDIT: I might have made a mistake and the right standard error in the second case might be $\frac{\sqrt{(X^TX)^{-1}_{jj}}\hat{\sigma}}{\sqrt{N}}$ Since $\hat{\sigma}$ estimates $\sigma$. But I still don't understand why the $\sqrt{N}$ is still in there.

Answer 1

It is better and clearer to take the following two steps to get the standard error of $\hat{\beta}_j$:

1. Compute the standard deviation of $\hat{\beta}_j$, which is the square root of $\operatorname{Var}(\hat{\beta}_j)$.

2. Estimate any unknown parameter(s) in the outcome of Step 1 to get the standard error of $\hat{\beta}_j$.

As can be seen from the above recipe, "standard deviation" and "standard error" of a statistic are two closely related, yet different concepts -- while standard deviation contains (typically unknown) parameters, standard error does not.

Since $\operatorname{Var}(\hat{\beta}) = \sigma^2(X^\top X)^{-1}$, we have $\operatorname{Var}(\hat{\beta}_j) = \sigma^2(X^\top X)^{-1}_{jj}$, whence $\operatorname{sd}(\hat{\beta}_j) = \sigma\sqrt{(X^\top X)^{-1}_{jj}}$. This completes Step 1.

Clearly, the $\sigma$ in $\operatorname{sd}(\hat{\beta}_j)$ is unknown and needs to be estimated, an obvious estimator of $\sigma$ is $\hat{\sigma}$. After substituting $\hat{\sigma}$ into $\sigma$ in the expression of $\operatorname{sd}(\hat{\beta}_j)$, we get $\operatorname{s.\!e.}(\hat{\beta}_j) = \hat{\sigma}\sqrt{(X^\top X)^{-1}_{jj}}$.

PS, in my opinion, I don't think using "z-score" to refer the statistic $\frac{\hat{\beta}_j}{\hat{\sigma}\sqrt{(X^\top X)^{-1}_{jj}}}$ an accurate and conventional choice, a better name should be "t-score". "z-score" is more appropriate when the parameter $\sigma$ is known to us, and the exact distribution of a "z-score" is Gaussian, as opposed to the $t$ distribution (as in this case).

Addendum

To OP's follow-up question:

I still don't understand (and want to understand) how this relates to the recipe of the standard error $\sigma/\sqrt{N}$.

the answer is easy: because this "recipe" only applies to a simpler model (i.e., $Y_1, \ldots, Y_n$ are drawn independently from a common distribution with mean $\mu$ and variance $\sigma^2$), and when the estimand/parameter is $\mu$, but not to a general estimator $\hat{\theta}$ of a parameter $\theta$ (in particular, using $\hat{\beta}_j$ to estimate $\beta_j$ in the regression setting). The correct way to get the standard error of a general $\hat{\theta}$ is completing the two steps outlined above, that is, you have to clearly derive the variance of $\hat{\theta}$, in this case $\hat{\beta}_j$. If you went through this process, you would find there is no place for the factor $\frac{1}{\sqrt{N}}$ in the expression.

Interestingly, the expression $\operatorname{s.\!e.}(\overline{Y}) = \frac{\hat{\sigma}}{\sqrt{N}}$ is a special case of the more general standard error formula $\operatorname{s.\!e.}(\hat{\beta}_j) = \hat{\sigma}\sqrt{(X^\top X)^{-1}_{jj}}$. Try to link them by embedding i.i.d. $\{Y_1, \ldots, Y_n\}$ into a simple linear regression setting. The hint is that the design matrix is \begin{align*} X = \begin{bmatrix} 1 \\ 1 \\ \vdots \\ 1 \end{bmatrix}, \end{align*} and there is only one $\beta$ coefficient $\beta_1$, i.e., the intercept. So you kind of reversed the reasoning order -- you could derive $\operatorname{s.\!e.}(\overline{Y}) = \frac{\hat{\sigma}}{\sqrt{N}}$ from $\operatorname{s.\!e.}(\hat{\beta}_j) = \hat{\sigma}\sqrt{(X^\top X)^{-1}_{jj}}$, but not the other way around.