# Exercises concerning a linear regression model with parameter $\beta$

I'm trying to solve the following exercises. I am unsure of what I'm doing so I appreciate any help.

My attempt:

(a) We have that

\begin{align*} \boldsymbol{\hat \beta} &= (\boldsymbol X^T \boldsymbol X)^{-1} \boldsymbol X^T \boldsymbol Y\\ &= \left(\begin{pmatrix} x_1 & x_2 & \cdots & x_n \end{pmatrix} \begin{pmatrix} x_1\\ x_2\\ \vdots\\ x_n \end{pmatrix}\right)^{-1} \begin{pmatrix} x_1 & x_2 & \cdots & x_n \end{pmatrix} \begin{pmatrix} Y_1\\ Y_2\\ \vdots\\ Y_n \end{pmatrix}\\ &= \Big(\sum_{i = 1}^n x_i^2\Big)^{-1} \Big(\sum_{i = 1}^n x_i Y_i\Big)\\ &= \frac{\sum_{i = 1}^n x_i Y_i}{\sum_{i = 1}^n x_i^2} \end{align*}

as desired. (Note that $$\boldsymbol X^T \boldsymbol X$$ is automatically invertible since it is just a (presumably non-zero) real number.)

(b) This estimator is linear because

\begin{align*} \tilde{\beta} &= \frac{1}{n} \begin{pmatrix} \frac{1}{x_1} & \frac{1}{x_2} & \cdots & \frac{1}{x_n} \end{pmatrix} \begin{pmatrix} Y_1\\ Y_2\\ \vdots\\ Y_n \end{pmatrix}, \end{align*}

that is, it can be written as a linear map.

It is unbiased because $$\text E (\tilde{\beta}) = \frac{1}{n} \sum_{i = 1}^n \frac{1}{x_i} \text E(Y_i) = \frac{1}{n} \sum_{i = 1}^n \frac{1}{x_i} \beta x_i = \frac{1}{n} n\beta = \beta$$.

Regarding variance, for $$\hat \beta$$ we have

\begin{align*} \text{Var}(\hat \beta) &= \text{Var}\Bigg( \frac{\sum_{i = 1}^n x_i Y_i}{\sum_{i = 1}^n x_i^2} \Bigg)\\ &= \frac{1}{\Big( \sum_{i = 1}^n x_i^2 \Big)^2} \text{Var} \Big( \sum_{i = 1}^n x_i Y_i \Big)\\ &= \frac{1}{\Big( \sum_{i = 1}^n x_i^2 \Big)^2} \sum_{i = 1}^n x_i^2 \text{Var}(Y_i)\\ &= \frac{\sigma^2}{\sum_{i = 1}^n x_i^2}. \end{align*}

For $$\tilde \beta$$, we have

\begin{align*} \text{Var}(\tilde \beta) &= \text{Var} \Bigg( \frac{1}{n} \sum_{i = 1}^n \frac{Y_i}{x_i} \Bigg)\\ &= \frac{1}{n^2} \sum_{i = 1}^n \text{Var} \Big( \frac{Y_i}{x_i} \Big)\\ &= \frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2} \text{Var} (Y_i)\\ &= \frac{\sigma^2}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2}. \end{align*}

Unless I've made mistakes to this point, I now need to compare the coefficients of $$\sigma^2$$ in both cases, i.e. compare $$\frac{1}{\sum_{i = 1}^n x_i^2}$$ with $$\frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2}$$, and hopefully see that the latter is at least as large as the former. I am not sure how to do this... maybe I need to try to use an inequality like Cauchy-Schwarz or something.

Edit: Following Glen_b's suggestion, I've used an arithmetic mean/harmonic mean inequality to show

\begin{align*} \frac{\sum_{i = 1}^n x_i^2}{n} \geq \frac{n}{\sum_{i = 1}^n \frac{1}{x_i^2}}\\ \Rightarrow \frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2} \geq \frac{1}{\sum_{i = 1}^n x_i^2} \end{align*}

as desired.

(c) I am not sure what to do here. Do I have to be able to say something like $$\text{Var}(\epsilon_i) =$$ [something], and $$\text{Cov}(\epsilon_i, \epsilon_j) =$$ [something]?

Thank you.

• Consider applying an inequality for arithmetic vs harmonic means to the $x_i^2$ values. Commented Sep 23, 2021 at 0:30
• @Glen_b I've followed your hint and it worked. Great suggestion. Thanks. If you have any suggestions for the last part, I'm all ears. Commented Sep 23, 2021 at 0:45
• Now ask yourself how I might have spotted that the inequality would be relevant. On c, consider trying a simple generalization first. If you had variances $v_i=1/w_i$, but retained independence, what would the BLUE look like! Commented Sep 23, 2021 at 0:55
• @Glen_b Now ask yourself...'' Presumably you are more experienced than I am so you saw the connection. Mean inequalities aren't really part of my toolkit. Regarding (c), you mean like if we had a new error vector $\boldsymbol{\epsilon}'$ where $\text{Var}(\epsilon'_i) = 1/\text{Var}(\epsilon_i) = 1/\sigma^2$? I guess the Gauss-Markov Theorem would still hold in this case. I have to think about it. Commented Sep 23, 2021 at 1:07
• Actually what I suggested is probably wrong, because I don't think that would change the BLUE at all. The Gauss-Markov theorem would still hold. I wonder if I need to look more closely at the definition of $\tilde \beta$ for clues... and maybe part (b) could be relevant here too. Commented Sep 23, 2021 at 1:15

I assume the $$x_i$$'s are fixed constants.

Consider uncorrelated but heteroscedastic errors , i.e.

$$\operatorname{Cov}(\varepsilon_i,\varepsilon_j)= \begin{cases}\sigma_i^2 &,\text{ if }i=j \\ 0 &, \text{ if }i\ne j \end{cases}$$

Assuming of course $$\sigma_i>0$$ for every $$i$$, rewrite the model as

$$\frac{Y_i}{\sigma_i}=\beta \frac{x_i}{\sigma_i}+\frac{\varepsilon_i}{\sigma_i} \quad\text{ or }, \quad Y_i^*=\beta x_i^*+\varepsilon_i^*$$

This model with $$Y_i^*$$ as response and $$x_i^*$$ as covariate satisfies the Gauss-Markov assumptions, so that OLS can be performed on this transformed model. This leads to the weighted least squares estimator of $$\beta$$:

$$\tilde \beta=\frac{\sum_{i=1}^n x_i^* Y_i^*}{\sum_{i=1}^n (x_i^*)^2}=\frac{\sum\limits_{i=1}^n \frac{x_i Y_i}{\sigma_i^2}}{\sum\limits_{i=1}^n \frac{x_i^2}{\sigma_i^2}}$$

If you choose $$\sigma_i^2 \propto x_i^2$$, i.e. $$\sigma_i^2=\sigma^2 x_i^2$$ for some positive $$\sigma$$, you get the estimator in your post:

$$\tilde \beta=\frac1n\sum_{i=1}^n \frac{Y_i}{x_i}$$