5
$\begingroup$

Consider the simple linear regression model $$y_i = \alpha + \beta x_i + u_i$$ with classic Gauss-Markov assumptions. In proving that $\hat{\beta}$, the OLS estimator for $\beta$, is the best linear unbiased estimator, one approach is to define an alternative estimator as a weighted sum of $y_i$: $$\tilde{\beta} = \sum_{i=1}^n c_i y_i$$ Then, we define $c_i = k_i + d_i$, where $k_i = \frac{x_i - \bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2}$ and so the OLS estimator for $\beta$ can be written in the form $\hat{\beta} = \sum_{i=1}^n k_i y_i$. To show that $\hat{\beta}$ is BLUE, the alternative estimator can be written as: $$\tilde{\beta} = \hat{\beta} + \sum_{i=1}^n d_i y_i$$ Hence, its variance can be written: $$Var(\tilde{\beta}) = Var(\hat{\beta}) + \sum_{i=1}^n d_i^2 Var(y_i) + 2\sum_{i=1}^n k_i d_i Var(y_i)$$ Then: \begin{align*} \sum_{i=1}^n k_i d_i &= \sum_{i=1}^n k_i(c_i - k_i) \\ &= \sum_{i=1}^n k_i c_i - \sum_{i=1}^n k_i^2 \\ &= \sum_{i=1}^n c_i \bigg(\frac{x_i - \bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2} \bigg) - \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} \\ &= \bigg(\frac{\sum_{i=1}^n c_i x_i - \bar{x} \sum_{i=1}^n c_i}{\sum_{i=1}^n (x_i - \bar{x})^2} \bigg) - \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} \end{align*} By the conditions of linearity and unbiasedness, it can be shown that $\sum_{i=1}^n c_i x_i = 1$, and $\sum_{i=1}^n c_i = 0$ - so: \begin{align*} \sum_{i=1}^n k_i d_i &= \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} - \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} = 0 \end{align*} The third term in the expression for $Var(\tilde{\beta})$ drops out. Then it is plain that the variance of any alternative unbiased estimator, $\tilde{\beta}$, for $\beta$ has a variance at least as large as $\hat{\beta}$: so the OLS estimator is BLUE. I want to prove that $\hat{\alpha}$, the OLS estimator for the intercept $\alpha$, is BLUE in the same way, but I'm having difficulty determining what value to now assign to $k_i$ such that $\hat{\alpha} = \sum_{i=1}^n k_i y_i$. So far, what I have is that: \begin{align*} \hat{\alpha} &= \bar{y} - \hat{\beta} \bar{x} \\ &= \frac{1}{n} \sum_{i=1}^n y_i - \frac{\sum_{i=1}^n (x_i - \bar{x})y_i}{\sum{i=1}^n (x_i - \bar{x})^2} \bar{x} \\ &= \sum_{i=1}^n y_i \bigg[\frac{1}{n} - \frac{(x_i - \bar{x})\bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2} \bigg]\\ &= \sum_{i=1}^n k_i y_i \end{align*} where $k_i = \frac{1}{n} - \frac{(x_i - \bar{x})\bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2}$, but things seem to go awry when I work through the rest of the proof.

$\endgroup$
1

2 Answers 2

7
$\begingroup$

This is one of those theorems that is easier to prove in greater generality using vector algebra than it is to prove with scalar algebra. To do this, consider the multiple linear regression model $\mathbf{Y} = \mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$ and consider the general linear estimator:

$$\hat{\boldsymbol{\beta}}_\mathbf{A} = \hat{\boldsymbol{\beta}}_\text{OLS} + \mathbf{A} \mathbf{Y} = [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] \mathbf{Y}.$$

Since the OLS estimator is unbiased and $\mathbb{E}(\mathbf{Y}) = \mathbf{x} \boldsymbol{\beta}$ this general linear estimator has bias:

$$\begin{align} \text{Bias}(\hat{\boldsymbol{\beta}}_\mathbf{A}, \boldsymbol{\beta}) &\equiv \mathbb{E}(\hat{\boldsymbol{\beta}}_\mathbf{A}) - \boldsymbol{\beta} \\[6pt] &= \mathbb{E}(\hat{\boldsymbol{\beta}}_\text{OLS} + \mathbf{A} \mathbf{Y}) - \boldsymbol{\beta} \\[6pt] &= \boldsymbol{\beta} + \mathbf{A} \mathbf{x} \boldsymbol{\beta} - \boldsymbol{\beta} \\[6pt] &= \mathbf{A} \mathbf{x} \boldsymbol{\beta}, \\[6pt] \end{align}$$

and so the requirement of unbiasedness imposes the restriction that $\mathbf{A} \mathbf{x} = \mathbf{0}$. The variance of the general linear estimator is:

$$\begin{align} \mathbb{V}(\hat{\boldsymbol{\beta}}_\mathbf{A}) &= \mathbb{V}(\mathbf{A} \mathbf{Y}) \\[6pt] &= [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] \mathbb{V}(\mathbf{Y}) [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}]^\text{T} \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}]^\text{T} \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] [\mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} \mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} \mathbf{A}^\text{T} + \mathbf{A} \mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} + (\mathbf{x}^\text{T} \mathbf{x})^{-1} (\mathbf{A} \mathbf{x})^\text{T} + (\mathbf{A} \mathbf{x}) (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} + (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{0}^\text{T} + \mathbf{0} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}]. \\[6pt] \end{align}$$

Hence, we have:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}_\mathbf{A}) - \mathbb{V}(\hat{\boldsymbol{\beta}}_\text{OLS}) = \sigma^2 \mathbf{A} \mathbf{A}^\text{T}.$$

Now, since $\mathbf{A} \mathbf{A}^\text{T}$ is a positive definite matrix, we can see that the variance of the general linear estimator is minimised when $\mathbf{A} = \mathbf{0}$, which yields the OLS estimator.

$\endgroup$
2
  • $\begingroup$ Thanks for that Ben. I was wondering if you have any pointers for how to complete the proof in scalar form - examiners can be tricky and I've seen past paper questions which ask for the proof that OLS estimators are BLUE to be given specifically in scalar form, with the examiner's report stating that students who give the matrix form proof drop marks for it. Unhelpfully, all the lecture notes I come across cover the full proof for $\beta$, the slope coefficient, but leave the proof for $\alpha$ as an exercise for the reader... $\endgroup$
    – greggs
    Jan 7, 2021 at 12:29
  • 1
    $\begingroup$ If it were me, I would simply not comply with the examiners' narrow and useless instruction, and take the resulting penalty in marks, taking solace in the knowledge that I am learning to do mathematics properly, without imposition of arbitrary constraint. $\endgroup$
    – Ben
    Jan 7, 2021 at 13:00
3
$\begingroup$

I eventually figured out where I was going wrong - so I'm going to post my work here in case anyone else gets stuck down the same rabbit hole. Start by defining an alternative estimator: $$\tilde{\alpha} = \sum_{i=1}^n c_i y_i$$ and define $c_i = k_i + d_i$, where $k_i$ are the weights on the OLS estimator $\hat{\alpha}$, so $k_i = \Big[\frac{1}{n} - \frac{(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}\bar{x}\Big]$. Then: $$\hat{\alpha} = \sum_{i=1}^n k_i y_i = \frac{1}{n} \sum_{i=1}^n y_i - \frac{\sum_{i=1}^n (x_i - \bar{x})y_i}{\sum_{i=1}^n (x_i - \bar{x})^2}\bar{x} = \bar{y} - \hat{\beta} \bar{x}$$ Hence, our new alternative estimator is: $$\tilde{\alpha} = \sum_{i=1}^n k_i y_i + \sum_{i=1}^n d_i y_i = \hat{\alpha} + \sum_{i=1}^n d_i y_i$$ So far so good. The next step is to find $Var(\tilde{\alpha})$: $$Var(\tilde{\alpha}) = Var\Big(\sum_{i=1}^n k_i y_i + \sum_{i=1}^n d_i y_i\Big) = Var\Big(\sum_{i=1}^n k_i y_i\Big) + \sum_{i=1}^n d_i^2 Var(y_i) + 2\sum_{i=1}^n k_i d_i Var(y_i)$$ Alternatively, $$Var(\tilde{\alpha}) = Var(\hat{\alpha}) + \sum_{i=1}^n d_i^2 Var(y_i) + 2\sum_{i=1}^n k_i d_i Var(y_i)$$ So, to show that $\hat{\alpha}$ is at least as efficient as any alternative (linear unbiased) estimator, i.e. $Var(\hat{\alpha}) \leq Var(\tilde{\alpha})$, we want to show that this third term drops out. Here's where I got tripped up: before, we did this with the condition of unbiasedness for $\beta$, which gave the conditions that $\sum_{i=1}^n c_i = 0$ and $\sum_{i=1}^n c_i x_i = 1$. But since we require that our new estimator is unbiased for $\alpha$, we get a different set of conditions. We need $\mathbb{E}(\tilde{\alpha}) = \alpha$, so: \begin{align*} \mathbb{E}(\tilde{\alpha}) &= \mathbb{E}\Big(\sum_{i=1}^n c_i y_i\Big) \\ &= \mathbb{E}\Big(\alpha \sum_{i=1}^n c_i + \beta \sum_{i=1}^n c_i x_i + \sum_{i=1}^n c_i u_i\Big) \end{align*} Since we want to keep the first term and we require the rest of the expression to drop out (to yield $\mathbb{E}(\tilde{\alpha}) = \alpha$), unbiasedness now imposes the condition that $\sum_{i=1}^n c_i = 1$ and $\sum_{i=1}^n c_i x_i = 0$: the opposite of the conditions for $\beta$. With these conditions updated, we can now show that: \begin{align*} \sum_{i=1}^n k_i d_i &= \sum_{i=1}^n k_i (c_i - k_i) \\ &= \sum_{i=1}^n k_i c_i - \sum_{i=1}^n k_i^2 \\ &= \Big[\frac{1}{n} \sum_{i=1}^n c_i - \frac{\bar{x} \sum_{i=1}^n c_i x_i - \bar{x}^2 \sum_{i=1}^n c_i}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] - \Big[\frac{n}{n^2} + \frac{\sum_{i=1}^n (x_i - \bar{x})^2 \bar{x}^2}{[\sum_{i=1}^n (x_i - \bar{x})^2]^2} - 2 \frac{\bar{x}^2 - \bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] \\ &= \Big[\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] - \Big[\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] \\ &= 0 \end{align*} So, now we have shown that: $$Var(\tilde{\alpha}) = Var(\hat{\alpha}) + \sum_{i=1}^n d_i^2 Var(y_i)$$ And since we've got a sum of squares (positive) multiplied by the variance of $y_i$ (also positive), this is sufficient to conclude that $Var(\tilde{\alpha}) \geq Var(\hat{\alpha})$, or in other words, $\hat{\alpha}$ is BLUE. As Ben's answer says, it's certainly quicker to give the proof with vector algebra - but in case anyone else has tricky examiners, then here's the scalar proof.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.