Suppose that we are carrying out a linear regression, in which we have $p$-parameters and $N$ observations in our training data. Then let us denote by $X$ the matrix of dimension $N\times (p+1)$ with each row and input vector where we have concatenated $1$ to the first position. Similarly let $y$ be the $N$-vector that has the outputs from the training data.
Now we are looking for the parameter $\beta$ in a model of the form:
$$Y=X^T\beta$$
and we know that our estimates, with respect to the RSS, are given by:
$$\hat{\beta} =(X^TX)^{-1}X^Ty$$
and then by making the assumption that our model is correct and that the errors are additive and Gaussian then we know that:
$$\hat{B} \sim N(\beta, (X^TX)^{-1}\sigma^2)$$
These assumption then allow us to test the significance of the parameter $\beta$ by using the test statistic:
$$z_j=\frac{\hat{\beta}}{\hat{\sigma}\sqrt{v_j}}$$
where $v_j$ is the diagonal entry of $(X^TX)^{-1}$
Now I am slightly confused as to what it means for us to test a parameter and find it not to be significant. Suppose that we test a ${\beta_j}$ and do not reject the null hypothesis that ${\beta_j}=0$, and so would drop this from our model. Will this not increase the RSS of the model that we have minimized in our parameter estimates?
I realize that my question is pretty ill-formed but I have gotten myself a bit confused: basically I am asking what it means for a parameter to be judged to be not significant with respect to minimizing the RSS of the model.
