5
$\begingroup$

Suppose I have some data $(x_i,y_i)$. If we perform ordinary least squares, we can get standard errors of the slope and intercept using estimates like $\hat{Var}(\hat{\beta}) = \hat{\sigma}^2 (X^\top X)^{-1}$ (see here and here).

However, suppose each $y_i$ came with a standard error $s_i$, possibly from some previous procedure. How would we obtain standard errors of the ordinary least squares slope and intercept that account for the standard errors from $s_i$? Or should we use a different regression technique?

$\endgroup$
2
  • $\begingroup$ Do you mean, you have some regression model $y_i = \mathbf{x}_i' \mathbf{\beta} + \epsilon_i$ and for whatever reason, you know that the variance of $\epsilon_i$ is $\sigma^2_i$, that is, (i) the variance of the residual is different for different observations (ii) you know what the variance for each observation is? Is this correct? $\endgroup$ Oct 8, 2016 at 4:19
  • $\begingroup$ @MatthewGunn Yes, I think that is what I mean, up to the fact that $s_i^2$ may only be an estimate of the true $\sigma_i^2$. But we can temporarily consider the simpler case where $\sigma_i^2$ is known for now. $\endgroup$
    – angryavian
    Oct 8, 2016 at 4:32

1 Answer 1

1
$\begingroup$

Assume regression model: $$ y_i = \mathbf{x}'_i \boldsymbol{\beta} + \epsilon_i$$

Let $\boldsymbol{\epsilon}$ be your vector of error terms.

If you know that $ \mathrm{Var}\left(\boldsymbol{\epsilon} \right) = \Omega$, for example:

$$ \Omega = \begin{bmatrix} \sigma^2_1 & 0 & 0& \ldots&0\\0 &\sigma^2_2 & 0 & \ldots&0\\ 0 & 0 &\sigma^2_3& \ldots & 0\\\ldots&\ldots&\ldots &\ldots&0\\0&0&0&0&\sigma^2_n\end{bmatrix} $$ Then you can more efficiently estimate $\boldsymbol{\beta}$ using generalized least squares.

The estimator $\hat{\mathbf{b}}$ for GLS is given by:

$$\hat{\mathbf{b}} = \left(X'\Omega^{-1} X \right)^{-1}\left(X'\Omega^{-1} \mathbf{y} \right) $$

The basic idea with GLS is to give observations that are more precisely observed higher weight. The danger of this approach of course is that if $\Omega$ is not correct, you can end up with something far worse than the equal weighting of regular OLS.

Note also that weighted least squares (as what would occur for this $\Omega$), is a special case of GLS.

If you just want to use OLS estimation but calculate standard errors assuming you know $\mathrm{Var}\left(\boldsymbol{\epsilon}\right)$

\begin{align*} \mathrm{Var}\left( \hat{\mathbf{b}}_{OLS} \right) &= \mathrm{Var}\left( \left(X'X \right)^{-1}X'\left(X\mathbf{\beta} + \boldsymbol{\epsilon} \right) \right)\\ &=\left(X'X \right)^{-1}X' \mathrm{Var}\left( \boldsymbol{\epsilon} \right) X \left(X'X \right)^{-1} \\ &=\left(X'X \right)^{-1}X' \Omega X \left(X'X \right)^{-1} \end{align*}

The variance of the GLS estimator is given by:

\begin{align*} \mathrm{Var}\left( \hat{\mathbf{b}}_{GLS} \right) &= \mathrm{Var}\left( \left(X'\Omega^{-1} X \right)^{-1}X'\Omega^{-1}\left(X\mathbf{\beta} + \boldsymbol{\epsilon} \right) \right)\\ &=\left(X'\Omega^{-1}X \right)^{-1}X'\Omega^{-1} \mathrm{Var}\left( \boldsymbol{\epsilon} \right) \Omega^{-1} X \left(X'\Omega^{-1}X \right)^{-1} \\ &=\left(X'\Omega^{-1}X \right)^{-1}X'\Omega^{-1} \Omega\Omega^{-1} X \left(X'\Omega^{-1}X \right)^{-1} \\ &=\left(X'\Omega^{-1}X \right)^{-1}X'\Omega^{-1} X \left(X'\Omega^{-1}X \right)^{-1}\\ &=\left(X'\Omega^{-1}X \right)^{-1} \end{align*}

$\endgroup$
5
  • $\begingroup$ Is the variance of the weighted least squares estimator the following? $$ Var(\hat{b}) = (X^\top \Omega^{-1} X)^{-1} X^\top \Omega^{-1} Var(\epsilon) \Omega^{-1} X(X^\top \Omega^{-1} X)^{-1} = (X^\top \Omega^{-1} X)^{-1} $$ $\endgroup$
    – angryavian
    Oct 8, 2016 at 5:02
  • $\begingroup$ @angryavian Yes, my typos should be fixed and the variance is given in my answer now too. $\endgroup$ Oct 8, 2016 at 5:11
  • 1
    $\begingroup$ I think if you have a "square root" so that $\Omega^{-1}=W^TW$, then I think the GLS problem for $(X,y,\Omega)$ is equivalent to an OLS problem for $(\hat{X},\hat{y})=(WX,Wy)$. $\endgroup$
    – GeoMatt22
    Oct 8, 2016 at 5:15
  • $\begingroup$ @GeoMatt22 Yes, it seems to be discussed on the Wikipedia page. $\endgroup$
    – angryavian
    Oct 8, 2016 at 5:16
  • $\begingroup$ @matthew I had some typos of my own as well, sorry. Thank you so much for your thorough explanations/references! Very much appreciated. $\endgroup$
    – angryavian
    Oct 8, 2016 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.