Exercises concerning a linear regression model with parameter $\beta$

Question

I'm trying to solve the following exercises. I am unsure of what I'm doing so I appreciate any help.

My attempt:

(a) We have that

\begin{align*} \boldsymbol{\hat \beta} &= (\boldsymbol X^T \boldsymbol X)^{-1} \boldsymbol X^T \boldsymbol Y\\ &= \left(\begin{pmatrix} x_1 & x_2 & \cdots & x_n \end{pmatrix} \begin{pmatrix} x_1\\ x_2\\ \vdots\\ x_n \end{pmatrix}\right)^{-1} \begin{pmatrix} x_1 & x_2 & \cdots & x_n \end{pmatrix} \begin{pmatrix} Y_1\\ Y_2\\ \vdots\\ Y_n \end{pmatrix}\\ &= \Big(\sum_{i = 1}^n x_i^2\Big)^{-1} \Big(\sum_{i = 1}^n x_i Y_i\Big)\\ &= \frac{\sum_{i = 1}^n x_i Y_i}{\sum_{i = 1}^n x_i^2} \end{align*}

as desired. (Note that $\boldsymbol X^T \boldsymbol X$ is automatically invertible since it is just a (presumably non-zero) real number.)

(b) This estimator is linear because

\begin{align*} \tilde{\beta} &= \frac{1}{n} \begin{pmatrix} \frac{1}{x_1} & \frac{1}{x_2} & \cdots & \frac{1}{x_n} \end{pmatrix} \begin{pmatrix} Y_1\\ Y_2\\ \vdots\\ Y_n \end{pmatrix}, \end{align*}

that is, it can be written as a linear map.

It is unbiased because $\text E (\tilde{\beta}) = \frac{1}{n} \sum_{i = 1}^n \frac{1}{x_i} \text E(Y_i) = \frac{1}{n} \sum_{i = 1}^n \frac{1}{x_i} \beta x_i = \frac{1}{n} n\beta = \beta$.

Regarding variance, for $\hat \beta$ we have

\begin{align*} \text{Var}(\hat \beta) &= \text{Var}\Bigg( \frac{\sum_{i = 1}^n x_i Y_i}{\sum_{i = 1}^n x_i^2} \Bigg)\\ &= \frac{1}{\Big( \sum_{i = 1}^n x_i^2 \Big)^2} \text{Var} \Big( \sum_{i = 1}^n x_i Y_i \Big)\\ &= \frac{1}{\Big( \sum_{i = 1}^n x_i^2 \Big)^2} \sum_{i = 1}^n x_i^2 \text{Var}(Y_i)\\ &= \frac{\sigma^2}{\sum_{i = 1}^n x_i^2}. \end{align*}

For $\tilde \beta$, we have

\begin{align*} \text{Var}(\tilde \beta) &= \text{Var} \Bigg( \frac{1}{n} \sum_{i = 1}^n \frac{Y_i}{x_i} \Bigg)\\ &= \frac{1}{n^2} \sum_{i = 1}^n \text{Var} \Big( \frac{Y_i}{x_i} \Big)\\ &= \frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2} \text{Var} (Y_i)\\ &= \frac{\sigma^2}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2}. \end{align*}

Unless I've made mistakes to this point, I now need to compare the coefficients of $\sigma^2$ in both cases, i.e. compare $\frac{1}{\sum_{i = 1}^n x_i^2}$ with $\frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2}$, and hopefully see that the latter is at least as large as the former. I am not sure how to do this... maybe I need to try to use an inequality like Cauchy-Schwarz or something.

Edit: Following Glen_b's suggestion, I've used an arithmetic mean/harmonic mean inequality to show

\begin{align*} \frac{\sum_{i = 1}^n x_i^2}{n} \geq \frac{n}{\sum_{i = 1}^n \frac{1}{x_i^2}}\\ \Rightarrow \frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2} \geq \frac{1}{\sum_{i = 1}^n x_i^2} \end{align*}

as desired.

(c) I am not sure what to do here. Do I have to be able to say something like ``$\text{Var}(\epsilon_i) = $ [something], and $\text{Cov}(\epsilon_i, \epsilon_j) = $ [something]?

Thank you.

Answer 1

I assume the $x_i$'s are fixed constants.

Consider uncorrelated but heteroscedastic errors , i.e.

$$\operatorname{Cov}(\varepsilon_i,\varepsilon_j)= \begin{cases}\sigma_i^2 &,\text{ if }i=j \\ 0 &, \text{ if }i\ne j \end{cases}$$

Assuming of course $\sigma_i>0$ for every $i$, rewrite the model as

$$\frac{Y_i}{\sigma_i}=\beta \frac{x_i}{\sigma_i}+\frac{\varepsilon_i}{\sigma_i} \quad\text{ or }, \quad Y_i^*=\beta x_i^*+\varepsilon_i^*$$

This model with $Y_i^*$ as response and $x_i^*$ as covariate satisfies the Gauss-Markov assumptions, so that OLS can be performed on this transformed model. This leads to the weighted least squares estimator of $\beta$:

$$\tilde \beta=\frac{\sum_{i=1}^n x_i^* Y_i^*}{\sum_{i=1}^n (x_i^*)^2}=\frac{\sum\limits_{i=1}^n \frac{x_i Y_i}{\sigma_i^2}}{\sum\limits_{i=1}^n \frac{x_i^2}{\sigma_i^2}}$$

If you choose $\sigma_i^2 \propto x_i^2$, i.e. $\sigma_i^2=\sigma^2 x_i^2$ for some positive $\sigma$, you get the estimator in your post:

$$\tilde \beta=\frac1n\sum_{i=1}^n \frac{Y_i}{x_i}$$