In machine learning and linear regression applications, the z-value is given by the following formula:

$$z_j = \frac{\beta_j}{\hat{\sigma}\sqrt{v_j}}$$


$$\hat{\sigma}^2 = \frac{1}{N-p-1}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2$$

and $\beta_j$ are the regression coefficients and $\hat{\sigma}^2$ is the variance of the coefficients; $v_j$ are the diagonal elements of $(X^TX)^{-1}$.

The denominator $\hat{\sigma}\sqrt{v_j}$is generally called standard error: why is that? Why is it considered an error and why standard? What does it tell/represent?

  • In OLS, your parameters are normally distributed. So the standard error is essentially the standard deviation of the sampling distribution of each parameter. Therefore, the standard error tells you how far on average sample parameters tend to deviate from the mean population parameter.

  • Then note, that when you divide each parameter by its standard error, you determine how statistically significant each parameter is. You noted the case of z-transformation, routinely applied as part of the Generalised Linear Models in R. The z-transformation assumes that population parameters are known. However, this is often NOT the case.

  • Therefore, in smaller samples especially, it is advisable to use t-transformation instead, as it will yield correct standard errors.

  • Having said that, in large samples, the difference between using z and t statistic for determining statistical significance is negligible, as both tend to converge.

> Related posts you will most definitely find enlightening:

Difference between Standard Error and Standard Deviation

Standard Error in Linear Regression

  • $\begingroup$ Thanks for your answer. Just few clarifications: in OLS my parameters are normally distributed because of the assumptions of Gaussian noise and additive model right? I know the z-score is more general and the numerator should be $\beta-\mu$ where $\mu$ is the mean. The z-score in thi specific caseis used as a method of feature selection: it is assuming that coefficient is 0 and (i.e., the feature can be ignored) and then check if data support this assumption. $\endgroup$ – Francesco Boi Dec 10 '19 at 14:43
  • $\begingroup$ From your second point, the z-score seems to be equivalent to a Signal-To-Noise-Ratio measure in the sense that, if the noise is too strong compared to the coefficient, then we are not able to use that feature significantly for estimation: is this totally wrong? $\endgroup$ – Francesco Boi Dec 10 '19 at 14:47
  • $\begingroup$ @FrancescoBoi please use the following thread it gives a very concise explanation as to why parameter estimates are normally distributed: stats.stackexchange.com/questions/254655/… $\endgroup$ – PsychometStats Dec 10 '19 at 15:19
  • $\begingroup$ @FrancescoBoi my apologies I find it difficult to understand what is meant by the Signal-to-Noise Ratio in relation to z-statistic. To give you a very intuitive meaning: suppose you have a $b_1$ parameter, then when you divide it by its associated s.e. you essentially transform the parameter into a known z-distribution. So you scale your parameter as a z-distribution score. Then, if the z values is < - 1.96 or > 1.96, your coefficient is non-significant at the 95% alpha level. Those are two small tails corresponding to the 5% of the distribution $\endgroup$ – PsychometStats Dec 10 '19 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.