7
$\begingroup$

This question uses the derivations found here.

The short version

Consider a regression model. If the error variance is a known function of the data (rather than a constant), under what conditions can we draw conclusions about the OLS estimates?

The long version

Notation

Denote:

  • $X = \left[\matrix{ X_{11} & \dots & X_{1p} \\ \vdots & \ddots & \vdots \\ X_{n1} & \dots & X_{np} \\ }\right]$
  • $\beta = \left(\beta_1, \dots, \beta_p\right)$
  • $Y = \left(Y_1, \dots, Y_n\right)$
  • $\epsilon = \left(\epsilon_1, \dots, \epsilon_n\right)$

Assume:

  • $Y= X \beta + \epsilon$
  • $\operatorname{E}\left(\epsilon\,|\,X\right)=0$ so that $E(Y\,|\,X) = X \beta$
  • $\operatorname{Var}\left(\epsilon\right)$ is diagonal.
  • $X$ is deterministic so we can drop the "$\left(\cdot\,|\,X\right)$".

Define:

  • $\hat{\beta}$: the OLS estimate of $\beta$ in the model $Y=X \beta + \epsilon$
  • $\tilde{\beta}$: an arbitrary competing estimate $\tilde{\beta} = A'Y$
  • $B = X \left(X'X\right)^{-1}$

Background

We derive $\operatorname{Var}\left(\hat{\beta}\right)$ by assuming that $\operatorname{E}\left(\epsilon\epsilon'\right) = \sigma^2 I$. Then we can conclude that: $$\begin{align} \operatorname{Var}\left(\hat{\beta}\right) &= \left(X'X\right)^{-1} X' \underbrace{\operatorname{E}\left(\epsilon\epsilon'\right)}_{=\sigma^2 I} X \left(X'X\right)^{-1} \\ &= \sigma^2 \left(X'X\right)^{-1} X' X \left(X'X\right)^{-1} \\ &= \sigma^2 \left(X'X\right)^{-1} \\ \end{align}$$

This in turn is used to show that $\hat{\beta}$ is efficient among unbiased estimators: $$\begin{align} \operatorname{Var}\left(\tilde{\beta}\right) - \operatorname{Var}\left(\hat{\beta}\right) &= \sigma^2 A'A - \sigma^2 \left(X'X\right)^{-1} \\ &= \sigma^2 A' M A \\ &\geq 0 \end{align}$$

The question

What if $\operatorname{Var}\left(\epsilon\right) = h\left(X\right)$ for a known function $h$?

This leaves us with $$ \operatorname{Var}\left(\hat{\beta}\right) = B' h\left(X\right) B $$ which is nice, but $$ \operatorname{Var}\left(\tilde{\beta}\right) - \operatorname{Var}\left(\hat{\beta}\right) = A' h\left(X\right) A - B' h\left(X\right) B $$ doesn't tell us anything.

What conditions on $h$ will allow us to learn something about $\operatorname{Var}\left(\hat{\beta}\right)$ and $\operatorname{Var}\left(\tilde{\beta}\right) - \operatorname{Var}\left(\hat{\beta}\right)$? Or (as per AdamO's comment) about the relative efficiency?

For instance, this reduces to generalized least squares when $h(X) = X' \Omega X$. But I'm mainly still interested in the case (as per the assumptions at the beginning) where $h(X)$ is diagonal.

Similarly, consider $$ h\left(X\right) = \left[\matrix{f(X_1 \cdot \beta_1) & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & f(X_p \cdot \beta_p) \\}\right] $$

where $f(z) = z$ (implied if $\epsilon$ is Poisson) or $f(z) \propto z^2$ (implied if $\epsilon$ is lognormal or gamma). This looks suspiciously like iteratively reweighted least squares.

$\endgroup$
  • $\begingroup$ In $\operatorname{E}(\epsilon\epsilon')=\sigma^2$, is the left hand side a matrix and the right hand side a scalar? Also, in $\operatorname{E}(\epsilon\epsilon')=h(X)$, can you still say observations are still iid? $\endgroup$ – Richard Hardy Mar 27 '15 at 18:48
  • $\begingroup$ @RichardHardy whoops, good point. Sloppy notation, I'll edit. $\endgroup$ – shadowtalker Mar 27 '15 at 18:54
  • $\begingroup$ I wonder what you expect to learn about $\operatorname{Var}\left(\hat{\beta}\right)$ and $\operatorname{Var}\left(\tilde{\beta}\right) - \operatorname{Var}\left(\hat{\beta}\right)$? That the latter is negative semidefinite or the like? Also, I am still not comfortable with iid in the iid case ... where $h(X)$ is diagonal. But perhaps I am just not getting it. $\endgroup$ – Richard Hardy Mar 27 '15 at 19:43
  • $\begingroup$ Do you know that $\mbox{Cov}(Y_i, Y_j) = 0$ for $i \neq j$? $\endgroup$ – AdamO Mar 27 '15 at 19:45
  • $\begingroup$ @RichardHardy that would be worth knowing. Can we derive bounds on it? Does it depend on $X$? The motivation for this question was originally "what happens if we fit a GLM with OLS?" but it took on a life of its own once I started writing out equations. $\endgroup$ – shadowtalker Mar 27 '15 at 19:46
8
$\begingroup$

It's an easy derivation to show that the least squares estimator:

$$ \hat{\beta} = \left( \mathbf{X}^T\mathbf{X} \right)^{-1} \mathbf{X}^T Y $$

has variance:

$$ \mbox{var} \left(\hat{\beta} \right)= \left( \mathbf{X}^T\mathbf{X} \right)^{-1} \mathbf{X}^T \mbox{var} \left(Y\right)\mathbf{X} \left( \mathbf{X}^T\mathbf{X} \right)^{-1} $$

If $h(X)$ is known then the inverse variance weighted least squares estimator: $(X^T W X)^{-1} X^T W Y$ is unbiased and efficient where $W = diag(h(X)^{-1})$.

The variance of the WLS estimator becomes:

$$ \mbox{var} (\hat{\beta}_{wls}) = (X^T W X)^{-1}$$

It's easy to show that if the mean model is correctly specified the unweighted version of OLS is NOT BIASED. It's NOT BIASED. It's NOT BIASED. -- that always bears repeating as many people don't understand: weighting here only gives you better efficiency.

How much better?

The relative efficiency of the two estimators is not to hard to work out, but WLS is uniformly better. Seber and Lee would have more details if you're interested.

$\endgroup$
  • $\begingroup$ Nothing like reinventing the wheel! I never quite understood WLS before now. Also, thanks for the book reference. $\endgroup$ – shadowtalker Mar 27 '15 at 20:05
  • $\begingroup$ @ssdecontrol, was this essentially what you were looking for? $\endgroup$ – Richard Hardy Mar 27 '15 at 20:06
  • $\begingroup$ @RichardHardy yes $\endgroup$ – shadowtalker Mar 27 '15 at 20:32
  • 2
    $\begingroup$ +1 Well, there's at least a little something more than merely better efficiency; if $h()$ is known, those more efficient weighted estimates also have correct small-sample conditional CIs and PIs and so on. $\endgroup$ – Glen_b Mar 28 '15 at 3:57
  • 1
    $\begingroup$ @Glen_b I did neglect to mention that, and that's entirely correct. Those conclusions are implied when we state that both OLS and WLS are unbiased estimators (when the mean model is correctly specified), but the WLS is more efficient. $\endgroup$ – AdamO Mar 30 '15 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.