2
$\begingroup$

I'm trying to solve the following exercises. I am unsure of what I'm doing so I appreciate any help.

enter image description here enter image description here

My attempt:

(a) We have that

\begin{align*} \boldsymbol{\hat \beta} &= (\boldsymbol X^T \boldsymbol X)^{-1} \boldsymbol X^T \boldsymbol Y\\ &= \left(\begin{pmatrix} x_1 & x_2 & \cdots & x_n \end{pmatrix} \begin{pmatrix} x_1\\ x_2\\ \vdots\\ x_n \end{pmatrix}\right)^{-1} \begin{pmatrix} x_1 & x_2 & \cdots & x_n \end{pmatrix} \begin{pmatrix} Y_1\\ Y_2\\ \vdots\\ Y_n \end{pmatrix}\\ &= \Big(\sum_{i = 1}^n x_i^2\Big)^{-1} \Big(\sum_{i = 1}^n x_i Y_i\Big)\\ &= \frac{\sum_{i = 1}^n x_i Y_i}{\sum_{i = 1}^n x_i^2} \end{align*}

as desired. (Note that $\boldsymbol X^T \boldsymbol X$ is automatically invertible since it is just a (presumably non-zero) real number.)

(b) This estimator is linear because

\begin{align*} \tilde{\beta} &= \frac{1}{n} \begin{pmatrix} \frac{1}{x_1} & \frac{1}{x_2} & \cdots & \frac{1}{x_n} \end{pmatrix} \begin{pmatrix} Y_1\\ Y_2\\ \vdots\\ Y_n \end{pmatrix}, \end{align*}

that is, it can be written as a linear map.

It is unbiased because $\text E (\tilde{\beta}) = \frac{1}{n} \sum_{i = 1}^n \frac{1}{x_i} \text E(Y_i) = \frac{1}{n} \sum_{i = 1}^n \frac{1}{x_i} \beta x_i = \frac{1}{n} n\beta = \beta$.

Regarding variance, for $\hat \beta$ we have

\begin{align*} \text{Var}(\hat \beta) &= \text{Var}\Bigg( \frac{\sum_{i = 1}^n x_i Y_i}{\sum_{i = 1}^n x_i^2} \Bigg)\\ &= \frac{1}{\Big( \sum_{i = 1}^n x_i^2 \Big)^2} \text{Var} \Big( \sum_{i = 1}^n x_i Y_i \Big)\\ &= \frac{1}{\Big( \sum_{i = 1}^n x_i^2 \Big)^2} \sum_{i = 1}^n x_i^2 \text{Var}(Y_i)\\ &= \frac{\sigma^2}{\sum_{i = 1}^n x_i^2}. \end{align*}

For $\tilde \beta$, we have

\begin{align*} \text{Var}(\tilde \beta) &= \text{Var} \Bigg( \frac{1}{n} \sum_{i = 1}^n \frac{Y_i}{x_i} \Bigg)\\ &= \frac{1}{n^2} \sum_{i = 1}^n \text{Var} \Big( \frac{Y_i}{x_i} \Big)\\ &= \frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2} \text{Var} (Y_i)\\ &= \frac{\sigma^2}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2}. \end{align*}

Unless I've made mistakes to this point, I now need to compare the coefficients of $\sigma^2$ in both cases, i.e. compare $\frac{1}{\sum_{i = 1}^n x_i^2}$ with $\frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2}$, and hopefully see that the latter is at least as large as the former. I am not sure how to do this... maybe I need to try to use an inequality like Cauchy-Schwarz or something.

Edit: Following Glen_b's suggestion, I've used an arithmetic mean/harmonic mean inequality to show

\begin{align*} \frac{\sum_{i = 1}^n x_i^2}{n} \geq \frac{n}{\sum_{i = 1}^n \frac{1}{x_i^2}}\\ \Rightarrow \frac{1}{n^2} \sum_{i = 1}^n \frac{1}{x_i^2} \geq \frac{1}{\sum_{i = 1}^n x_i^2} \end{align*}

as desired.

(c) I am not sure what to do here. Do I have to be able to say something like ``$\text{Var}(\epsilon_i) = $ [something], and $\text{Cov}(\epsilon_i, \epsilon_j) = $ [something]?

Thank you.

$\endgroup$
5
  • $\begingroup$ Consider applying an inequality for arithmetic vs harmonic means to the $x_i^2$ values. $\endgroup$
    – Glen_b
    Sep 23, 2021 at 0:30
  • $\begingroup$ @Glen_b I've followed your hint and it worked. Great suggestion. Thanks. If you have any suggestions for the last part, I'm all ears. $\endgroup$
    – Novice
    Sep 23, 2021 at 0:45
  • $\begingroup$ Now ask yourself how I might have spotted that the inequality would be relevant. On c, consider trying a simple generalization first. If you had variances $v_i=1/w_i$, but retained independence, what would the BLUE look like! $\endgroup$
    – Glen_b
    Sep 23, 2021 at 0:55
  • $\begingroup$ @Glen_b ``Now ask yourself...'' Presumably you are more experienced than I am so you saw the connection. Mean inequalities aren't really part of my toolkit. Regarding (c), you mean like if we had a new error vector $\boldsymbol{\epsilon}'$ where $\text{Var}(\epsilon'_i) = 1/\text{Var}(\epsilon_i) = 1/\sigma^2$? I guess the Gauss-Markov Theorem would still hold in this case. I have to think about it. $\endgroup$
    – Novice
    Sep 23, 2021 at 1:07
  • $\begingroup$ Actually what I suggested is probably wrong, because I don't think that would change the BLUE at all. The Gauss-Markov theorem would still hold. I wonder if I need to look more closely at the definition of $\tilde \beta$ for clues... and maybe part (b) could be relevant here too. $\endgroup$
    – Novice
    Sep 23, 2021 at 1:15

1 Answer 1

0
$\begingroup$

I assume the $x_i$'s are fixed constants.

Consider uncorrelated but heteroscedastic errors , i.e.

$$\operatorname{Cov}(\varepsilon_i,\varepsilon_j)= \begin{cases}\sigma_i^2 &,\text{ if }i=j \\ 0 &, \text{ if }i\ne j \end{cases}$$

Assuming of course $\sigma_i>0$ for every $i$, rewrite the model as

$$\frac{Y_i}{\sigma_i}=\beta \frac{x_i}{\sigma_i}+\frac{\varepsilon_i}{\sigma_i} \quad\text{ or }, \quad Y_i^*=\beta x_i^*+\varepsilon_i^*$$

This model with $Y_i^*$ as response and $x_i^*$ as covariate satisfies the Gauss-Markov assumptions, so that OLS can be performed on this transformed model. This leads to the weighted least squares estimator of $\beta$:

$$\tilde \beta=\frac{\sum_{i=1}^n x_i^* Y_i^*}{\sum_{i=1}^n (x_i^*)^2}=\frac{\sum\limits_{i=1}^n \frac{x_i Y_i}{\sigma_i^2}}{\sum\limits_{i=1}^n \frac{x_i^2}{\sigma_i^2}}$$

If you choose $\sigma_i^2 \propto x_i^2$, i.e. $\sigma_i^2=\sigma^2 x_i^2$ for some positive $\sigma$, you get the estimator in your post:

$$\tilde \beta=\frac1n\sum_{i=1}^n \frac{Y_i}{x_i}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.