3
$\begingroup$

In ordinary least squares, for example in an experimental design case, I obtain the regression coefficents by:

$ \hat B = {({X^t}{X})}^{-1}X^ty$

Then, my null hypothesis for each coefficent is:

$H_0 : \hat B_i = 0$

$H_1 : \hat B_i \neq 0$

Next, by assuming the coefficents follows $t$ distribution:

$t_{calculated} = \dfrac {\hat B_i}{se({\hat B_i})}$ where $se({\hat B_i})$ is the standard error of coefficent $\hat B_i$

Again, for OLS, I keep reading that $se({\hat B_i})$ is square root of the corresponding diagonal element ($i^{th}$ diagonal element) of ${\hat \sigma^2}{({X^t}{X})}^{-1}$ where ${\hat \sigma^2}$ is mean square error and is defined as $\sum(y-\hat y)^2$

My questions are:

  1. What if I obtain my regression coefficients by other regression methods such as Ridge, LASSO, ElasticNet, Partial Least Squares etc..? Is this test and/or the way of calculating $t$ value is still valid? I am suspecting it is not, because there are regression techniques which do not involve the calculation of ${({X^t}{X})}^{-1}$ at all.

  2. Why and how do we assume ${\hat \sigma^2}{({X^t}{X})}^{-1}$ is standard error of regression coefficients $B$ in the first place? I think I don't understand the concept of standard error of a regression coefficient in first place.

Small note which may be relevant: I am dealing with a matrix having multicollinearity and OLS fails badly. I think the multicollinearity issue adds another dimension to significance tests and makes things even more complicated.

$\endgroup$
  • 1
    $\begingroup$ here is a step in the right direction bmcbioinformatics.biomedcentral.com/articles/10.1186/… $\endgroup$ – Jacob H Jul 5 '17 at 21:25
  • $\begingroup$ Also remember your coefficient estimates are biased $\endgroup$ – Jacob H Jul 5 '17 at 21:25
  • $\begingroup$ Do you mean that I am omitting intercept term? $\endgroup$ – theGD Jul 5 '17 at 21:39
  • $\begingroup$ No. Shrinkage estimators (like ridge and lasso) generate biased coefficients. An OLS estimator is BLUE, therefore a Ridge regression is shrinking the unbiased OLS estimator toward zero. Therefore, if you're attempting to use the Ridge Regression for inference you need to be careful. For inference, you're better dealing with the multicollinearity head on, i.e. remove highly correlated variables, use PCA to combine several highly correlated variables, etc.... IMO shrinkage model are best utilized to forecast/predict. However, if you ok with some bias, go ahead. $\endgroup$ – Jacob H Jul 5 '17 at 22:15
  • 1
    $\begingroup$ Rob Tibshirani published a paper, A significance test for the lasso, about this topic. There is also a presentation, and this covariance test is implemented in R package covTest. $\endgroup$ – schrodingercat Nov 16 '18 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.