9
$\begingroup$

In ordinary least squares, for example in an experimental design case, I obtain the regression coefficents by:

$ \hat B = {({X^t}{X})}^{-1}X^ty$

Then, my null hypothesis for each coefficent is:

$H_0 : \hat B_i = 0$

$H_1 : \hat B_i \neq 0$

Next, by assuming the coefficents follows $t$ distribution:

$t_{calculated} = \dfrac {\hat B_i}{se({\hat B_i})}$ where $se({\hat B_i})$ is the standard error of coefficent $\hat B_i$

Again, for OLS, I keep reading that $se({\hat B_i})$ is square root of the corresponding diagonal element ($i^{th}$ diagonal element) of ${\hat \sigma^2}{({X^t}{X})}^{-1}$ where ${\hat \sigma^2}$ is mean square error and is defined as $\sum(y-\hat y)^2$

My questions are:

  1. What if I obtain my regression coefficients by other regression methods such as Ridge, LASSO, ElasticNet, Partial Least Squares etc..? Is this test and/or the way of calculating $t$ value is still valid? I am suspecting it is not, because there are regression techniques which do not involve the calculation of ${({X^t}{X})}^{-1}$ at all.

  2. Why and how do we assume ${\hat \sigma^2}{({X^t}{X})}^{-1}$ is standard error of regression coefficients $B$ in the first place? I think I don't understand the concept of standard error of a regression coefficient in first place.

Small note which may be relevant: I am dealing with a matrix having multicollinearity and OLS fails badly. I think the multicollinearity issue adds another dimension to significance tests and makes things even more complicated.

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ here is a step in the right direction bmcbioinformatics.biomedcentral.com/articles/10.1186/… $\endgroup$
    – Jacob H
    Jul 5, 2017 at 21:25
  • $\begingroup$ Also remember your coefficient estimates are biased $\endgroup$
    – Jacob H
    Jul 5, 2017 at 21:25
  • $\begingroup$ Do you mean that I am omitting intercept term? $\endgroup$
    – gunakkoc
    Jul 5, 2017 at 21:39
  • $\begingroup$ No. Shrinkage estimators (like ridge and lasso) generate biased coefficients. An OLS estimator is BLUE, therefore a Ridge regression is shrinking the unbiased OLS estimator toward zero. Therefore, if you're attempting to use the Ridge Regression for inference you need to be careful. For inference, you're better dealing with the multicollinearity head on, i.e. remove highly correlated variables, use PCA to combine several highly correlated variables, etc.... IMO shrinkage model are best utilized to forecast/predict. However, if you ok with some bias, go ahead. $\endgroup$
    – Jacob H
    Jul 5, 2017 at 22:15
  • 2
    $\begingroup$ Rob Tibshirani published a paper, A significance test for the lasso, about this topic. There is also a presentation, and this covariance test is implemented in R package covTest. $\endgroup$
    – schrodingercat
    Nov 16, 2018 at 11:31

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.