16
$\begingroup$

I realize that this is a very basic question, but I can't find an answer anywhere.

I'm computing regression coefficients using either the normal equations or QR decomposition. How can I compute standard errors for each coefficient? I usually think of standard errors as being computed as:

$SE_\bar{x}\ = \frac{\sigma_{\bar x}}{\sqrt{n}}$

What is $\sigma_{\bar x}$ for each coefficient? What is the most efficient way to compute this in the context of OLS?

$\endgroup$
18
$\begingroup$

When doing least squares estimation (assuming a normal random component) the regression parameter estimates are normally distributed with mean equal to the true regression parameter and covariance matrix $\Sigma = s^2\cdot(X^TX)^{-1}$ where $s^2$ is the residual variance and $X^TX$ is the design matrix. $X^T$ is the transpose of $X$ and $X$ is defined by the model equation $Y=X\beta+\epsilon$ with $\beta$ the regression parameters and $\epsilon$ is the error term. The estimated standard deviation of a beta parameter is gotten by taking the corresponding term in $(X^TX)^{-1}$ multiplying it by the sample estimate of the residual variance and then taking the square root. This is not a very simple calculation but any software package will compute it for you and provide it in the output.

Example

On page 134 of Draper and Smith (referenced in my comment), they provide the following data for fitting by least squares a model $Y = \beta_0 + \beta_1 X + \varepsilon$ where $\varepsilon \sim N(0, \mathbb{I}\sigma^2)$.

                      X                      Y                    XY
                      0                     -2                     0
                      2                      0                     0
                      2                      2                     4
                      5                      1                     5
                      5                      3                    15
                      9                      1                     9
                      9                      0                     0
                      9                      0                     0
                      9                      1                     9
                     10                     -1                   -10
                    ---                     --                   ---
Sum                  60                      5                    32
Sum of  Squares     482                     21                   528

Looks like an example where the slope should be close to 0.

$$X^t = \pmatrix{ 1 &1 &1 &1 &1 &1 &1 &1 &1 &1 \\ 0 &2 &2 &5 &5 &9 &9 &9 &9 &10 }.$$

So

$$X^t X = \pmatrix{n &\sum X_i \\ \sum X_i &\sum X_i^2} = \pmatrix{10 &60 \\60 &482}$$

and

$$\eqalign{ (X^t X)^{-1} &= \pmatrix{ \frac{\sum X_i^2}{n \sum (X_i - \bar{X})^2} &\frac{-\bar{X}}{\sum (X_i-\bar{X})^2} \\ \frac{-\bar{X}}{\sum (X_i-\bar{X})^2} &\frac{1}{\sum (X_i-\bar{X})^2} } \\ &= \pmatrix{\frac{482}{10(122)} &-\frac{6}{122} \\ -\frac{6}{122} &\frac{1}{122}} \\ &= \pmatrix{0.395 &-0.049 \\ -0.049 &0.008} }$$

where $\bar{X} = \sum X_i/n = 60/10 = 6$.

Estimate for $β = (X^TX)^{-1} X^TY$ = ( b0 ) =(Yb-b1 Xb) b1 Sxy/Sxx

b1 = 1/61 = 0.0163 and b0 = 0.5- 0.0163(6) = 0.402

From $(X^TX)^{-1}$ above Sb1 =Se (0.008) and Sb0=Se(0.395) where Se is the estimated standard deviation for the error term. Se =√2.3085.

Sorry that the equations didn't carry subscripting and superscripting when I cut and pasted them. The table didn't reproduce well either because the spaces got ignored. The first string of 3 numbers correspond to the first values of X Y and XY and the same for the followinf strings of three. After Sum comes the sums for X Y and XY respectively and then the sum of squares for X Y and XY respectively. The 2x2 matrices got messed up too. The values after the brackets should be in brackets underneath the numbers to the left.

$\endgroup$
  • 2
    $\begingroup$ Not meant as a plug for my book but i go through the computations of the least squares solution in simple linear regression (Y=aX+b) and calculate the standard errors for a and b, pp.101-103, The Essentials of Biostatistics for Physicians, Nurses, and Clinicians, Wiley 2011. a more detailed description can be found In Draper and Smith Applied Regression Analysis 3rd Edition, Wiley New York 1998 page 126-127. In my answer that follows I will take an example from Draper and Smith. $\endgroup$ – Michael Chernick May 7 '12 at 15:53
  • 8
    $\begingroup$ When I started interacting with this site, Michael, I had similar feelings. With experience, they have changed. It's worthwhile knowing some $\TeX$ and once you do, it's (almost) as fast to type it in as it is to type in anything in English. I also learned, by studying exemplary posts (such as many replies by @chl, cardinal, and other high-reputation-per-post users), that providing references, clear illustrations, and well-thought out equations is usually highly appreciated and well received. High quality is one thing distinguishing this site from most others. $\endgroup$ – whuber May 7 '12 at 21:19
  • 2
    $\begingroup$ That is all nice Bill and it is nice that so many people are dedicated to give those high quality posts. I may use Latex for other purposes, like publishing papers. But I don't have the time to go to all the effort that people expect of me on this site. i am not going to invest the time just to provide service on this site. $\endgroup$ – Michael Chernick May 7 '12 at 21:42
  • 4
    $\begingroup$ I think the disconnect is here: "This is just one of many things about this site that requires those posting to put in extra time and effort" - @whuber and I are both saying that it, in fact, does not take extra time if you know how to do it. We don't learn $\TeX$ so that we can post on this site - we (at least I) learn $\TeX$ because it's an important skill to have as a statistician and happens to make posts much more readable on this site. $\endgroup$ – Macro May 8 '12 at 11:05
  • 3
    $\begingroup$ Like many of the people on here, yes, I work as a statistician, but I also happen to find it fun - this site is recreational for me and it's a nice bonus that others find some of my posts useful. If you find marking up your equations with $\TeX$ to be work and don't think it's worth learning then so be it, but know that some of your content will be overlooked. $\endgroup$ – Macro May 8 '12 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.