Say I have n sample, each with p explanatory variables, namely X. I also have n corresponding response variables, namely y.

I want to understand the how the explanatory variables contribute to the response variable.

We have two classical approaches:

1). I can get the effect sizes by linear regression, like solving the following problem, $$ \arg\min_\beta ||y-X\beta||_2^2 $$ and then I can analyze the $\beta$ I got.

2). Alternatively, I can run p hypothesis testings with the ith null hypothesis states that the ith variable has nothing to do with y. And then I can analyze the p-values I got.

Then, for the 1st method, I lost the statistical guarantee because I don't have p-values. But for the 2nd method, I only test these explanatory variables independently, and I lost the interaction between them.

Is there a way that I can run the linear regression, get $\beta$ and then test the p-values for it?

Also, I know there are packages like R that can run linear regression and reports effect sizes together with p-values. But my question is more about how is it calculated? (For example, I need to know how it's calculated if I want to implement my own tool.)


Thanks for Matthew's comments about first calculating $\beta$'s variance with $(X^TX)^{-1}$, but what if this is a high dimension problem $p>n$, so that $(X^TX)$ cannot be inversed? (I assume most linear regression problems today will be high-dimensional ones? At least the one I'm facing at is.)

  • $\begingroup$ Perhaps duplicate of: stats.stackexchange.com/questions/44838/… That calculates standard errors under the assumption of homoskedasticity. Note there are different types of standard errors (eg. robust standard errors, clustered standard errors) based on different assumptions. $\endgroup$ – Matthew Gunn Dec 11 '16 at 5:09
  • $\begingroup$ @MatthewGunn Thanks. I am still confused because if I want to do it for all $\beta$ together, then $X^TX\sigma$ is a $p\times p$ matrix, but shouldn't the variance of $\beta$ a vector of length $p$? $\endgroup$ – Haohan Wang Dec 11 '16 at 5:14
  • 2
    $\begingroup$ $\hat{\mathrm{Var}}(\mathbf{b}) = \left( X^T X \right) ^{-1} \hat{\sigma}^2 $ is an estimated covariance matrix for your estimated coefficients $\mathbf{b}$. For example, $\mathrm{Var}(b_3)$ would be the 3rd element on the diagonal of the matrix, $\mathrm{Var}(b_4)$ would be the 4th element on the diagonal of the matrix etc... Note also that $\hat{\sigma}^2 = \frac{1}{n-p} \sum_i e_i^2$. $\endgroup$ – Matthew Gunn Dec 11 '16 at 6:06
  • $\begingroup$ @MatthewGunn Thank you very much. Is there anything I can do if $p$ is greater than $n$ (so $X^TX$ cannot be inversed? ) $\endgroup$ – Haohan Wang Dec 11 '16 at 19:57
  • 2
    $\begingroup$ On your last comment, if your number of regressors $k$ is greater than your number of observations $n$, I don't see how you can possibly get reliable p-values for individual coefficients. I don't know what you can do instead, and I'm not an expert on the high dimensional stuff. $\endgroup$ – Matthew Gunn Dec 11 '16 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.