Suppose I have some data $(x_i,y_i)$. If we perform ordinary least squares, we can get standard errors of the slope and intercept using estimates like $\hat{Var}(\hat{\beta}) = \hat{\sigma}^2 (X^\top X)^{-1}$ (see here and here).

However, suppose each $y_i$ came with a standard error $s_i$, possibly from some previous procedure. How would we obtain standard errors of the ordinary least squares slope and intercept that account for the standard errors from $s_i$? Or should we use a different regression technique?

  • $\begingroup$ Do you mean, you have some regression model $y_i = \mathbf{x}_i' \mathbf{\beta} + \epsilon_i$ and for whatever reason, you know that the variance of $\epsilon_i$ is $\sigma^2_i$, that is, (i) the variance of the residual is different for different observations (ii) you know what the variance for each observation is? Is this correct? $\endgroup$ – Matthew Gunn Oct 8 '16 at 4:19
  • $\begingroup$ @MatthewGunn Yes, I think that is what I mean, up to the fact that $s_i^2$ may only be an estimate of the true $\sigma_i^2$. But we can temporarily consider the simpler case where $\sigma_i^2$ is known for now. $\endgroup$ – angryavian Oct 8 '16 at 4:32

Assume regression model: $$ y_i = \mathbf{x}'_i \boldsymbol{\beta} + \epsilon_i$$

Let $\boldsymbol{\epsilon}$ be your vector of error terms.

If you know that $ \mathrm{Var}\left(\boldsymbol{\epsilon} \right) = \Omega$, for example:

$$ \Omega = \begin{bmatrix} \sigma^2_1 & 0 & 0& \ldots&0\\0 &\sigma^2_2 & 0 & \ldots&0\\ 0 & 0 &\sigma^2_3& \ldots & 0\\\ldots&\ldots&\ldots &\ldots&0\\0&0&0&0&\sigma^2_n\end{bmatrix} $$ Then you can more efficiently estimate $\boldsymbol{\beta}$ using generalized least squares.

The estimator $\hat{\mathbf{b}}$ for GLS is given by:

$$\hat{\mathbf{b}} = \left(X'\Omega^{-1} X \right)^{-1}\left(X'\Omega^{-1} \mathbf{y} \right) $$

The basic idea with GLS is to give observations that are more precisely observed higher weight. The danger of this approach of course is that if $\Omega$ is not correct, you can end up with something far worse than the equal weighting of regular OLS.

Note also that weighted least squares (as what would occur for this $\Omega$), is a special case of GLS.

If you just want to use OLS estimation but calculate standard errors assuming you know $\mathrm{Var}\left(\boldsymbol{\epsilon}\right)$

\begin{align*} \mathrm{Var}\left( \hat{\mathbf{b}}_{OLS} \right) &= \mathrm{Var}\left( \left(X'X \right)^{-1}X'\left(X\mathbf{\beta} + \boldsymbol{\epsilon} \right) \right)\\ &=\left(X'X \right)^{-1}X' \mathrm{Var}\left( \boldsymbol{\epsilon} \right) X \left(X'X \right)^{-1} \\ &=\left(X'X \right)^{-1}X' \Omega X \left(X'X \right)^{-1} \end{align*}

The variance of the GLS estimator is given by:

\begin{align*} \mathrm{Var}\left( \hat{\mathbf{b}}_{GLS} \right) &= \mathrm{Var}\left( \left(X'\Omega^{-1} X \right)^{-1}X'\Omega^{-1}\left(X\mathbf{\beta} + \boldsymbol{\epsilon} \right) \right)\\ &=\left(X'\Omega^{-1}X \right)^{-1}X'\Omega^{-1} \mathrm{Var}\left( \boldsymbol{\epsilon} \right) \Omega^{-1} X \left(X'\Omega^{-1}X \right)^{-1} \\ &=\left(X'\Omega^{-1}X \right)^{-1}X'\Omega^{-1} \Omega\Omega^{-1} X \left(X'\Omega^{-1}X \right)^{-1} \\ &=\left(X'\Omega^{-1}X \right)^{-1}X'\Omega^{-1} X \left(X'\Omega^{-1}X \right)^{-1}\\ &=\left(X'\Omega^{-1}X \right)^{-1} \end{align*}

| cite | improve this answer | |
  • $\begingroup$ Is the variance of the weighted least squares estimator the following? $$ Var(\hat{b}) = (X^\top \Omega^{-1} X)^{-1} X^\top \Omega^{-1} Var(\epsilon) \Omega^{-1} X(X^\top \Omega^{-1} X)^{-1} = (X^\top \Omega^{-1} X)^{-1} $$ $\endgroup$ – angryavian Oct 8 '16 at 5:02
  • $\begingroup$ @angryavian Yes, my typos should be fixed and the variance is given in my answer now too. $\endgroup$ – Matthew Gunn Oct 8 '16 at 5:11
  • 1
    $\begingroup$ I think if you have a "square root" so that $\Omega^{-1}=W^TW$, then I think the GLS problem for $(X,y,\Omega)$ is equivalent to an OLS problem for $(\hat{X},\hat{y})=(WX,Wy)$. $\endgroup$ – GeoMatt22 Oct 8 '16 at 5:15
  • $\begingroup$ @GeoMatt22 Yes, it seems to be discussed on the Wikipedia page. $\endgroup$ – angryavian Oct 8 '16 at 5:16
  • $\begingroup$ @matthew I had some typos of my own as well, sorry. Thank you so much for your thorough explanations/references! Very much appreciated. $\endgroup$ – angryavian Oct 8 '16 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.