Regression coefficients! Everybody loves 'em.[Citation needed]

You know what else everybody loves? Test statistics for regression coefficients![Disputed, quite disputed, but with pushback]

Test statistics for regression coefficients are standard in statistical software output (typically t or z test statistics, depending on the nature of the estimator), and are just dandy for testing null hypotheses of the form $H_{0}\text{: }\beta = 0$ vs $H_{A}\text{: }\beta\ne 0$. [Hot damn!]

However… what if we want to test null hypotheses of the form $H_{0}\text{: }\beta = c$ for $c \ne 0$?

I know that a t test for a Pearson's correlation coefficient is pretty straightforward when the null hypothesis being tested is of the form $H_{0}\text{: }\rho =0$ vs $H_{A}\text{: }\rho \ne 0$, but that the math gets wonkier when the null hypothesis being tested is of the form $H_{0}\text{: }\rho =c$ for $c\ne 0$. So I wonder…

How to calculate the t or z test statistic for the null hypothesis $\boldsymbol{H_{0}\textbf{: }\beta = c}$ for $\boldsymbol{c \ne 0}$? Is it as simple as $(\hat{\beta}-c)/\hat{\sigma}_{\beta}$ or does it get funky?

Note: This is not a question about a test of one regression coefficient equaling another regression coefficient (i.e. not a test of $H_{0}\text{: }\beta_{A} = \beta_{B}$).

Edit: I bring up Pearson's $\rho$ in motivating my question, because (in the OLS case) $\rho$ and $\beta$ share the same numerator ($\rho$ is just the numerator normalized over the sample size), but calculating a t statistic for $\rho$ gets funky as noted above. I assume this is because the standard error no longer has the form under the $c=0$ null, and wonder why the standard error of $\beta$ does not likewise get funky.

  • $\begingroup$ I know for sure that this has been covered a few times before in answers but for some reason I am having trouble turning them up at the moment. $\endgroup$ – Glen_b Mar 29 '19 at 0:02
  • 1
    $\begingroup$ @Glen_b Well... but have they ever been asked with such zesty panache? $\endgroup$ – Alexis Mar 29 '19 at 15:12

When testing $H_0:\rho=\rho_0\,(\ne0)$ against any suitable alternative, one usually resorts to the variance stabilising Fisher transformation of the sample correlation coefficient $r$.

The usual t-test you are referring to is mainly reserved for the case $\rho_0=0$.

For moderately large $n$ (for example when $n\ge 25$), we have

$$\sqrt{n-3}(\tanh^{-1}(r)-\tanh^{-1}(\rho))\stackrel{a}\sim N(0,1)$$

So the appropriate test statistic under $H_0$ would be

$$T=\sqrt{n-3}(\tanh^{-1}(r)-\tanh^{-1}(\rho_0))\stackrel{a}\sim N(0,1)$$

In simple linear regression with normality assumption of errors (having variances $\sigma^2$), we have the exact distribution of the least square estimator $\hat\beta$ of the slope $\beta$, given by $\frac{(\hat\beta-\beta)\sqrt{s_{xx}}}{\sigma}\sim N(0,1)$ where $s_{xx}=\sum (x_i-\bar x)^2$.

Now if $\sigma$ is known, we test $H_0:\beta=\beta_0$ using the statistic $$T_1=\frac{(\hat\beta-\beta_0)\sqrt{s_{xx}}}{\sigma}\stackrel{H_0}\sim N(0,1)$$

If $\sigma$ is not known we estimate $\sigma$ by the residual sd $s$, where $s^2=\frac{SSE}{n-2}$. The test statistic is now

$$T_2=\frac{(\hat\beta-\beta_0)\sqrt{s_{xx}}}{s}\stackrel{H_0}\sim t_{n-2}$$

The variance of $\hat\beta$ in both these cases is independent of $\beta$ (the parameter of interest), so the variance stabilising transformation on $r$ while testing '$\rho=\rho_0$' is not required here. Not to mention that the test for '$\rho=\rho_0$' is a large sample test, while those involving $\beta$ are not.

  • $\begingroup$ Thank you, StubborAtom. Can you help me understand why the variance of $\beta$ doesn't also need some kind of stabilizing transformation under a null such as the one I present, given that, for example in the OLS case, it shares the same numerator as $\rho$? $\endgroup$ – Alexis Mar 28 '19 at 5:43
  • 2
    $\begingroup$ @Alexis Are you talking about the slope $\beta$ in simple linear regression (with the normality assumption of residuals)? If that is the case, the test statistic involving $\beta$ has an exact distribution and not a large sample test as the one involving $\rho$. $\endgroup$ – StubbornAtom Mar 28 '19 at 6:02
  • 2
    $\begingroup$ Had the test been $H_0:\rho=0$ then it would be comparable with the test $:H_0:\beta=0$ because the former implies the latter under null. $\endgroup$ – StubbornAtom Mar 28 '19 at 6:08
  • $\begingroup$ "The variance of $\beta$ in both these cases is independent of $\beta$ (the parameter of interest), so the variance stabilising transformation on $r$ while testing '$\rho=\rho_{0}$' is not required here." Nailed it. Thank you! $\endgroup$ – Alexis Mar 28 '19 at 16:37
  • 1
    $\begingroup$ @Alexis Yes I just used $T_1,T_2$ to mean test statistic #1 for $\sigma$ known and test statistic #2 for $\sigma$ unknown. $\endgroup$ – StubbornAtom Jun 5 '20 at 16:44

If one is using something like a Wald statistic, a very simple way to do this to test if

$H_o: \beta - c = 0$ vs $H_a: \beta - c \neq 0$

Since we know that $\hat \beta \text{ } \dot \sim N(\beta, se)$, then $\hat \beta -c \text{ } \dot \sim N(\beta - c, se)$, since $c$ is just a constant. This gives us a Wald statistic of $ \frac{\hat \beta - c}{ se }$.

In the case that we are not using a Wald statistic, we can (hopefully, if the software allows) include an offset in our model. That is, we can simply add $cx^*$, where $x^*$ is the covariate of interest, directly into the linear predictor. Then, if we wanted to do something like a likelihood ratio test, we could have one model with no free parameter associated with $x^*$ and another one that includes a free $x^*\beta^*$ to be estimated.

  • $\begingroup$ Thanks Cliff AB! What does the dot above the $\sim$ mean? $\endgroup$ – Alexis Mar 28 '19 at 0:18
  • 1
    $\begingroup$ @Alexis: "Approximately distributed as", rather than "distributed as". Since Wald statistics rely on asymptotics. $\endgroup$ – Cliff AB Mar 28 '19 at 0:18
  • $\begingroup$ Thank you again! Ok, next question (possibly rolled into your answer). Why does the same logic not apply to the t test of Pearson's $\hat{\rho}$? Is it because $\rho$ only goes from $-1$ to $1$? $\endgroup$ – Alexis Mar 28 '19 at 0:21
  • 1
    $\begingroup$ @Alexis: ha, that's actually the much harder question, in my opinion. I think it's because one is not using a Wald test statistic (i.e., normal approximation)? I'm not super familiar with the ins and outs of Pearson tests. $\endgroup$ – Cliff AB Mar 28 '19 at 0:27
  • $\begingroup$ What is tripping me up and why I pitched Pearson's $\rho$ vs. $\beta$ is that they have the same numerators! For $H_{0}:\rho = 0$ the test statistic $t = \frac{\hat{\rho}}{\sigma_{\rho}}$, where the latter $= \sqrt{\frac{1-\hat{r}}{n-2}}$. I guess under the null, that SE changes dramatically for $\rho$, and am wondering why it does not change dramatically for $\beta$? $\endgroup$ – Alexis Mar 28 '19 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.