Prove that the OLS estimator of the intercept is BLUE

$$\newcommand{\Var}{\operatorname{Var}}$$Consider the simple linear regression model $$y_i = \alpha + \beta x_i + u_i$$ with classic Gauss-Markov assumptions. In proving that $$\hat{\beta}$$, the OLS estimator for $$\beta$$, is the best linear unbiased estimator, one approach is to define an alternative estimator as a weighted sum of $$y_i$$: $$\tilde{\beta} = \sum_{i=1}^n c_i y_i$$ Then, we define $$c_i = k_i + d_i$$, where $$k_i = \frac{x_i - \bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2}$$ and so the OLS estimator for $$\beta$$ can be written in the form $$\hat{\beta} = \sum_{i=1}^n k_i y_i$$. To show that $$\hat{\beta}$$ is BLUE, the alternative estimator can be written as: $$\tilde{\beta} = \hat{\beta} + \sum_{i=1}^n d_i y_i$$ Hence, its variance can be written: $$\Var(\tilde{\beta}) = \Var(\hat{\beta}) + \sum_{i=1}^n d_i^2 \Var(y_i) + 2\sum_{i=1}^n k_i d_i \Var(y_i)$$ Then: \begin{align*} \sum_{i=1}^n k_i d_i &= \sum_{i=1}^n k_i(c_i - k_i) \\ &= \sum_{i=1}^n k_i c_i - \sum_{i=1}^n k_i^2 \\ &= \sum_{i=1}^n c_i \bigg(\frac{x_i - \bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2} \bigg) - \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} \\ &= \bigg(\frac{\sum_{i=1}^n c_i x_i - \bar{x} \sum_{i=1}^n c_i}{\sum_{i=1}^n (x_i - \bar{x})^2} \bigg) - \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} \end{align*} By the conditions of linearity and unbiasedness, it can be shown that $$\sum_{i=1}^n c_i x_i = 1$$, and $$\sum_{i=1}^n c_i = 0$$ - so: \begin{align*} \sum_{i=1}^n k_i d_i &= \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} - \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} = 0 \end{align*} The third term in the expression for $$\Var(\tilde{\beta})$$ drops out. Then it is plain that the variance of any alternative unbiased estimator, $$\tilde{\beta}$$, for $$\beta$$ has a variance at least as large as $$\hat{\beta}$$: so the OLS estimator is BLUE. I want to prove that $$\hat{\alpha}$$, the OLS estimator for the intercept $$\alpha$$, is BLUE in the same way, but I'm having difficulty determining what value to now assign to $$k_i$$ such that $$\hat{\alpha} = \sum_{i=1}^n k_i y_i$$. So far, what I have is that: \begin{align*} \hat{\alpha} &= \bar{y} - \hat{\beta} \bar{x} \\ &= \frac{1}{n} \sum_{i=1}^n y_i - \frac{\sum_{i=1}^n (x_i - \bar{x})y_i}{\sum{i=1}^n (x_i - \bar{x})^2} \bar{x} \\ &= \sum_{i=1}^n y_i \bigg[\frac{1}{n} - \frac{(x_i - \bar{x})\bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2} \bigg]\\ &= \sum_{i=1}^n k_i y_i \end{align*} where $$k_i = \frac{1}{n} - \frac{(x_i - \bar{x})\bar{x}}{\sum_{i=1}^n (x_i - \bar{x})^2}$$, but things seem to go awry when I work through the rest of the proof.

• Commented Mar 27, 2021 at 18:02

This is one of those theorems that is easier to prove in greater generality using vector algebra than it is to prove with scalar algebra. To do this, consider the multiple linear regression model $$\mathbf{Y} = \mathbf{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$$ and consider the general linear estimator:

$$\hat{\boldsymbol{\beta}}_\mathbf{A} = \hat{\boldsymbol{\beta}}_\text{OLS} + \mathbf{A} \mathbf{Y} = [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] \mathbf{Y}.$$

Since the OLS estimator is unbiased and $$\mathbb{E}(\mathbf{Y}) = \mathbf{x} \boldsymbol{\beta}$$ this general linear estimator has bias:

\begin{align} \text{Bias}(\hat{\boldsymbol{\beta}}_\mathbf{A}, \boldsymbol{\beta}) &\equiv \mathbb{E}(\hat{\boldsymbol{\beta}}_\mathbf{A}) - \boldsymbol{\beta} \\[6pt] &= \mathbb{E}(\hat{\boldsymbol{\beta}}_\text{OLS} + \mathbf{A} \mathbf{Y}) - \boldsymbol{\beta} \\[6pt] &= \boldsymbol{\beta} + \mathbf{A} \mathbf{x} \boldsymbol{\beta} - \boldsymbol{\beta} \\[6pt] &= \mathbf{A} \mathbf{x} \boldsymbol{\beta}, \\[6pt] \end{align}

and so the requirement of unbiasedness imposes the restriction that $$\mathbf{A} \mathbf{x} = \mathbf{0}$$. The variance of the general linear estimator is:

\begin{align} \mathbb{V}(\hat{\boldsymbol{\beta}}_\mathbf{A}) &= \mathbb{V}([(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] \mathbf{Y}) \\[6pt] &= [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] \mathbb{V}(\mathbf{Y}) [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}]^\text{T} \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}]^\text{T} \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} + \mathbf{A}] [\mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} \mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T} \mathbf{A}^\text{T} + \mathbf{A} \mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} + (\mathbf{x}^\text{T} \mathbf{x})^{-1} (\mathbf{A} \mathbf{x})^\text{T} + (\mathbf{A} \mathbf{x}) (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} + (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{0}^\text{T} + \mathbf{0} (\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}] \\[6pt] &= \sigma^2 [(\mathbf{x}^\text{T} \mathbf{x})^{-1} + \mathbf{A} \mathbf{A}^\text{T}]. \\[6pt] \end{align}

Hence, we have:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}_\mathbf{A}) - \mathbb{V}(\hat{\boldsymbol{\beta}}_\text{OLS}) = \sigma^2 \mathbf{A} \mathbf{A}^\text{T}.$$

Now, since $$\mathbf{A} \mathbf{A}^\text{T}$$ is a positive definite matrix, we can see that the variance of the general linear estimator is minimised when $$\mathbf{A} = \mathbf{0}$$, which yields the OLS estimator.

• Thanks for that Ben. I was wondering if you have any pointers for how to complete the proof in scalar form - examiners can be tricky and I've seen past paper questions which ask for the proof that OLS estimators are BLUE to be given specifically in scalar form, with the examiner's report stating that students who give the matrix form proof drop marks for it. Unhelpfully, all the lecture notes I come across cover the full proof for $\beta$, the slope coefficient, but leave the proof for $\alpha$ as an exercise for the reader... Commented Jan 7, 2021 at 12:29
• If it were me, I would simply not comply with the examiners' narrow and useless instruction, and take the resulting penalty in marks, taking solace in the knowledge that I am learning to do mathematics properly, without imposition of arbitrary constraint.
– Ben
Commented Jan 7, 2021 at 13:00

I eventually figured out where I was going wrong - so I'm going to post my work here in case anyone else gets stuck down the same rabbit hole. Start by defining an alternative estimator: $$\tilde{\alpha} = \sum_{i=1}^n c_i y_i$$ and define $$c_i = k_i + d_i$$, where $$k_i$$ are the weights on the OLS estimator $$\hat{\alpha}$$, so $$k_i = \Big[\frac{1}{n} - \frac{(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}\bar{x}\Big]$$. Then: $$\hat{\alpha} = \sum_{i=1}^n k_i y_i = \frac{1}{n} \sum_{i=1}^n y_i - \frac{\sum_{i=1}^n (x_i - \bar{x})y_i}{\sum_{i=1}^n (x_i - \bar{x})^2}\bar{x} = \bar{y} - \hat{\beta} \bar{x}$$ Hence, our new alternative estimator is: $$\tilde{\alpha} = \sum_{i=1}^n k_i y_i + \sum_{i=1}^n d_i y_i = \hat{\alpha} + \sum_{i=1}^n d_i y_i$$ So far so good. The next step is to find $$Var(\tilde{\alpha})$$: $$Var(\tilde{\alpha}) = Var\Big(\sum_{i=1}^n k_i y_i + \sum_{i=1}^n d_i y_i\Big) = Var\Big(\sum_{i=1}^n k_i y_i\Big) + \sum_{i=1}^n d_i^2 Var(y_i) + 2\sum_{i=1}^n k_i d_i Var(y_i)$$ Alternatively, $$Var(\tilde{\alpha}) = Var(\hat{\alpha}) + \sum_{i=1}^n d_i^2 Var(y_i) + 2\sum_{i=1}^n k_i d_i Var(y_i)$$ So, to show that $$\hat{\alpha}$$ is at least as efficient as any alternative (linear unbiased) estimator, i.e. $$Var(\hat{\alpha}) \leq Var(\tilde{\alpha})$$, we want to show that this third term drops out. Here's where I got tripped up: before, we did this with the condition of unbiasedness for $$\beta$$, which gave the conditions that $$\sum_{i=1}^n c_i = 0$$ and $$\sum_{i=1}^n c_i x_i = 1$$. But since we require that our new estimator is unbiased for $$\alpha$$, we get a different set of conditions. We need $$\mathbb{E}(\tilde{\alpha}) = \alpha$$, so: \begin{align*} \mathbb{E}(\tilde{\alpha}) &= \mathbb{E}\Big(\sum_{i=1}^n c_i y_i\Big) \\ &= \mathbb{E}\Big(\alpha \sum_{i=1}^n c_i + \beta \sum_{i=1}^n c_i x_i + \sum_{i=1}^n c_i u_i\Big) \end{align*} Since we want to keep the first term and we require the rest of the expression to drop out (to yield $$\mathbb{E}(\tilde{\alpha}) = \alpha$$), unbiasedness now imposes the condition that $$\sum_{i=1}^n c_i = 1$$ and $$\sum_{i=1}^n c_i x_i = 0$$: the opposite of the conditions for $$\beta$$. With these conditions updated, we can now show that: \begin{align*} \sum_{i=1}^n k_i d_i &= \sum_{i=1}^n k_i (c_i - k_i) \\ &= \sum_{i=1}^n k_i c_i - \sum_{i=1}^n k_i^2 \\ &= \Big[\frac{1}{n} \sum_{i=1}^n c_i - \frac{\bar{x} \sum_{i=1}^n c_i x_i - \bar{x}^2 \sum_{i=1}^n c_i}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] - \Big[\frac{n}{n^2} + \frac{\sum_{i=1}^n (x_i - \bar{x})^2 \bar{x}^2}{[\sum_{i=1}^n (x_i - \bar{x})^2]^2} - 2 \frac{\bar{x}^2 - \bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] \\ &= \Big[\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] - \Big[\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \Big] \\ &= 0 \end{align*} So, now we have shown that: $$Var(\tilde{\alpha}) = Var(\hat{\alpha}) + \sum_{i=1}^n d_i^2 Var(y_i)$$ And since we've got a sum of squares (positive) multiplied by the variance of $$y_i$$ (also positive), this is sufficient to conclude that $$Var(\tilde{\alpha}) \geq Var(\hat{\alpha})$$, or in other words, $$\hat{\alpha}$$ is BLUE. As Ben's answer says, it's certainly quicker to give the proof with vector algebra - but in case anyone else has tricky examiners, then here's the scalar proof.

• Thanks for this nice proof (+1). I'm more familiar with scalar algebra than with matrix algebra. One probably stupid question. The unbiasedness of the alternative estimator puts constraints on the c(i)'s. I do not get why the expectation of c(i)u(i) drops of or is apparently zero. Ofcourse for the expectation of the u(i) this is obvious, but why for the sum of the products c(i)u(i)?
– BenP
Commented Jul 1 at 11:19
• @BenP My understanding is that $E[u_i]=0$ according to the stated Gauss-Markov assumptions regarding the model. Therefore $E[\sum_{i=1}^n c_i u_i ]=0$ Commented Jul 2 at 12:51
• @Roger_Jia Thanks, I see that now. The $c_i$ can be seen as constants chosen in a particular way, so that the we have $c_1*E(u_1) + c_2*E(u_2) + ... = 0$.
– BenP
Commented Jul 2 at 13:47

C&B offers a particularly ingenuous and elegant way to handle this in an elementary manner, the meat of which is in the following general result:

Result $$1:$$ Let $$\boldsymbol v$$ be an arbitrary vector of constants and let $$\boldsymbol c$$ be a vector of positive constants. Define $$\mathcal A:=\{\boldsymbol a\mid \sum a_i =0\}.$$ If $$\bar v_c$$ is the weighted mean of the elements of $$\boldsymbol v$$ with weights corresponding to the elements of $$\boldsymbol c,$$ then $$\sup_{\boldsymbol a\in\mathcal A}\left\{\frac{\left(\sum_{i=1}^k a_iv_i\right)^2}{\sum_{i=1}^k\frac{ a_i^2}{c_i}}\right\}= \sum_{i=1}^k c_i(v_i-\bar v_c) ^2.\tag 1\label 1$$ The supremum is attainable at any $$\boldsymbol a$$ where $$a_i:= Kc_i(v_i-\bar v_c), ~~K\in\mathbb R\setminus\{0\}.$$

The proof is based on Cauchy-Schwarz Inequality.

Consider a collection $$\mathcal B:=\left\{\boldsymbol b\mid \sum b_i=0\wedge \sum b_i^2/c_i=1\right\}.$$ For any $$\boldsymbol a\in\mathcal A,$$ take $$\boldsymbol b\in \mathcal B$$ with $$b_i:= \frac{a_i}{\sqrt{\sum_{i=1}^k a_i^2/c_i}}.$$ What is to be maximized is $$\left(\sum_{i=1}^k b_iv_i\right) ^2.$$

Define $$C:=\sum c_i.$$ Now consider two random variables $$B, V$$ such that $$\Pr\left(B=\frac{b_i}{c_i}, V=v_i\right) =\frac{c_i}C, ~~i\in\{1,\ldots,k\}.$$ It is easy to note that $$\mathbf EB=0, ~\mathbf EV=\bar v_c.$$ Then \begin{align}\left\{\sum_{i=1}^k \left(\frac{b_i}{c_i}\right)(v_i)\left(\frac{c_i}C\right)\right\}^2&=(\operatorname{Cov}(B,V))^2\\&\overset{\textrm{CS}}{\leq}(\operatorname{Var} B) (\operatorname{Var} V) \\&\leq \left(\sum_{i=1}^k \left(\frac{b_i}{c_i}\right)^2\left(\frac{c_i}C\right)\right)\left(\sum_{i=1}^k (v_i-\bar v_c) ^2\left(\frac{c_i}C\right)\right)\tag 2\label 2;\end{align} using $$\sum b_i^2/c_i=1$$ in $$\eqref 2,$$ it can be concluded that the RHS of $$\eqref 1$$ acts as an upper bound. Finally for the specific choice of $$a_i,$$ the attainment can be deduced.

$$\blacksquare$$

In the simple regression problem, any linear estimator would be of the form $$\sum_{i=1}^n d_iY_i.$$ Unbiasedness of the estimator would constrain $$\boldsymbol d$$ such that \begin{align}\sum_{i=1}^n d_i&=0,\\ \sum_{i=1}^n d_ix_i&= 1.\end{align}\tag 3\label 3

To obtain BLUE of $$\beta, ~\sum_{i=1}^n d_i^2$$ needs to be minimized for $$\operatorname{Var}\left(\sum_{i=1}^n d_iY_i\right)=\sigma^2\sum_{i=1}^n d_i^2.$$ This is how Result $$1$$ would be used: take $$k=n, ~v_i=x_i, ~c_i=1, ~a_i=d_i.$$ This means $$\bar v_c=\bar x.$$ If $$d_i:= K(x_i-\bar x),$$ then such $$\boldsymbol d$$ would maximize $$\left(\sum_{i=1}^n d_ix_i\right) ^2/\sum_{i=1}^n d_i^2$$ among all $$\boldsymbol d$$s satisfying $$\eqref 3.$$

Again by $$\eqref 3,~1=\sum_{i=1}^n d_ix_i= K\sum_{i=1}^n (x_i-\bar x) x_i= \sum_{i=1}^n (x_i-\bar x)^2\implies K=1/\sum_{i=1}^n (x_i-\bar x)^2.$$ Then one concludes $$d_i= \frac{(x_i-\bar x) }{\sum_{i=1}^n (x_i-\bar x)^2},~i=\{1,\ldots,n\}\tag 4\label 4.$$

The BLUE of $$\alpha$$ follows suit.

--

Reference:

$$\rm[I]$$ Statistical Inference, George Casella, Roger L. Berger, Wadsworth, $$2002,$$ sec. $$11.2.4, 11.3.2.$$

This is largely a reproduction of my earlier CV.SE post.