Say I have n sample, each with p explanatory variables, namely X. I also have n corresponding response variables, namely y.

I want to understand the how the explanatory variables contribute to the response variable.

We have two classical approaches:

1). I can get the effect sizes by linear regression, like solving the following problem, $$ \arg\min_\beta ||y-X\beta||_2^2 $$ and then I can analyze the $\beta$ I got.

2). Alternatively, I can run p hypothesis testings with the ith null hypothesis states that the ith variable has nothing to do with y. And then I can analyze the p-values I got.

Then, for the 1st method, I lost the statistical guarantee because I don't have p-values. But for the 2nd method, I only test these explanatory variables independently, and I lost the interaction between them.

Is there a way that I can run the linear regression, get $\beta$ and then test the p-values for it?

Also, I know there are packages like R that can run linear regression and reports effect sizes together with p-values. But my question is more about how is it calculated? (For example, I need to know how it's calculated if I want to implement my own tool.)


Thanks for Matthew's comments about first calculating $\beta$'s variance with $(X^TX)^{-1}$, but what if this is a high dimension problem $p>n$, so that $(X^TX)$ cannot be inversed? (I assume most linear regression problems today will be high-dimensional ones? At least the one I'm facing at is.)

  • $\begingroup$ Perhaps duplicate of: stats.stackexchange.com/questions/44838/… That calculates standard errors under the assumption of homoskedasticity. Note there are different types of standard errors (eg. robust standard errors, clustered standard errors) based on different assumptions. $\endgroup$ – Matthew Gunn Dec 11 '16 at 5:09
  • $\begingroup$ @MatthewGunn Thanks. I am still confused because if I want to do it for all $\beta$ together, then $X^TX\sigma$ is a $p\times p$ matrix, but shouldn't the variance of $\beta$ a vector of length $p$? $\endgroup$ – Haohan Wang Dec 11 '16 at 5:14
  • 2
    $\begingroup$ $\hat{\mathrm{Var}}(\mathbf{b}) = \left( X^T X \right) ^{-1} \hat{\sigma}^2 $ is an estimated covariance matrix for your estimated coefficients $\mathbf{b}$. For example, $\mathrm{Var}(b_3)$ would be the 3rd element on the diagonal of the matrix, $\mathrm{Var}(b_4)$ would be the 4th element on the diagonal of the matrix etc... Note also that $\hat{\sigma}^2 = \frac{1}{n-p} \sum_i e_i^2$. $\endgroup$ – Matthew Gunn Dec 11 '16 at 6:06
  • $\begingroup$ @MatthewGunn Thank you very much. Is there anything I can do if $p$ is greater than $n$ (so $X^TX$ cannot be inversed? ) $\endgroup$ – Haohan Wang Dec 11 '16 at 19:57
  • 2
    $\begingroup$ On your last comment, if your number of regressors $k$ is greater than your number of observations $n$, I don't see how you can possibly get reliable p-values for individual coefficients. I don't know what you can do instead, and I'm not an expert on the high dimensional stuff. $\endgroup$ – Matthew Gunn Dec 11 '16 at 21:42

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.