0
$\begingroup$

Suppose that we are carrying out a linear regression, in which we have $p$-parameters and $N$ observations in our training data. Then let us denote by $X$ the matrix of dimension $N\times (p+1)$ with each row and input vector where we have concatenated $1$ to the first position. Similarly let $y$ be the $N$-vector that has the outputs from the training data.

Now we are looking for the parameter $\beta$ in a model of the form:

$$Y=X^T\beta$$

and we know that our estimates, with respect to the RSS, are given by:

$$\hat{\beta} =(X^TX)^{-1}X^Ty$$

and then by making the assumption that our model is correct and that the errors are additive and Gaussian then we know that:

$$\hat{B} \sim N(\beta, (X^TX)^{-1}\sigma^2)$$

These assumption then allow us to test the significance of the parameter $\beta$ by using the test statistic:

$$z_j=\frac{\hat{\beta}}{\hat{\sigma}\sqrt{v_j}}$$

where $v_j$ is the diagonal entry of $(X^TX)^{-1}$

Now I am slightly confused as to what it means for us to test a parameter and find it not to be significant. Suppose that we test a ${\beta_j}$ and do not reject the null hypothesis that ${\beta_j}=0$, and so would drop this from our model. Will this not increase the RSS of the model that we have minimized in our parameter estimates?

I realize that my question is pretty ill-formed but I have gotten myself a bit confused: basically I am asking what it means for a parameter to be judged to be not significant with respect to minimizing the RSS of the model.

$\endgroup$
  • $\begingroup$ Tests are about parameters, not about estimates. So your statement $H_o: \hat \beta_j = 0$ does not make sense. $\endgroup$ – Michael M Oct 29 '15 at 16:55
  • $\begingroup$ @MichaelM Sorry, I got a little confused when I was typing this up, shouldn't have rushed so much! I have corrected so that my question makes (more) sense now. Thanks. $\endgroup$ – hmmmm Oct 29 '15 at 17:01
1
$\begingroup$

In layman's words, the $z$ test is about how far away your estimate lies from the hypothesised population mean (parameter). If the significance test shows that the distance is not far enough for the estimate to be significnatly different from 0 or hypothesised value, you should drop the variable from the model. Note that given your data, OLS attempts to minimise the RSS. Non-significance could be due to various issues in your model, i.e. mis-specification, multicollinearity, serial correlation or heteroskedastic error etc.

$\endgroup$
  • $\begingroup$ Thanks for the answer but I am still a little confused, sorry. I understand the purpose of the $z$-test but if we remove the non-significant parameter from the model this will increase the RSS right? As otherwise this parameter would just have been $0$? $\endgroup$ – hmmmm Oct 29 '15 at 18:32
  • $\begingroup$ Yes it does, RSS will go up marginally if the removed variable does not have a sig. coeff estimate. However, if it has a sig coef estimate, your RSS will go up by a larger amount when you drop it from the model. You are not losing much if you drop the insignificant coeff estimates $\endgroup$ – mr.rox Oct 29 '15 at 18:56
  • $\begingroup$ It is a bad advice to just drop insignificant coefficients from the model, as such strategies might bias the final results. Depending on the setting it might make perfect sense, but clearly not in general. $\endgroup$ – Michael M Oct 30 '15 at 7:00
  • $\begingroup$ @MichaelM yeah, one should not drop the insig coeff variable, but his question was about why drop if it minimises RSS. If your model passes all diagnostics tests i.e. serial correl, heteroskedasticity etc tests, it may be that your variable is redundant in the model or that if you have an underlying economic theory that underlies the model, keep the variable. The insignificance may stem from the sample size or sampling bias etc. $\endgroup$ – mr.rox Oct 30 '15 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.