Finding sampling distribution of normal MLE and likelihood

Question

I'm reviewing old exams in preparation for a statistics final, and I'm stuck on a particular question:

Suppose that you have n independent random variables $Y_i$, with each distributed normal with expected value $\beta x_i$ and known variance $\sigma^2$. Further suppose that the $x_i$ are known constant values, and define the likelihood as $\mathcal{L}(\beta; y) = \frac{f(y ; \beta)}{f(y ; \hat{\beta})}$, where $\hat{\beta}$ is the maximum likelihood estimate for $\beta$.

1. Show that $\hat{\beta} = \frac{\sum_{i=1}^n x_i y_i}{\sum_{i=1}^n x_i^2}$
2. Find the sampling distribution of the MLE
3. Show that the sampling distribution of $-2\ln(\mathcal{L}(\beta; y))$ is Chi-squared with 1 df

I had no problem with (1), but I'm unsure about (2) and (3).

For (2), I rewrote (1) as $\sum_{i=1}^n k_i y_i$ where the $k_i$ are constants and equal to $\frac{x_i}{\sum_{i=1}^n x_i^2}$. I then used linearity of expectation to say that the expected value of the sampling distribution is $\sum_{i=1}^n k_i \beta x_i$. For the variance, I believe it would be $\mathrm{Var}(\sum_{i=1}^n k_i y_i) = \sum_{i=1}^n \mathrm{Var} (k_i y_i)$ because of independence. Therefore, the variance of the sampling distribution should be $\sigma^2 \sum_{i=1}^n k_i^2$. Since the $y_i$'s are normal, their sum should be normal, so the sampling distribution is $N(\sum_{i=1}^n k_i \beta x_i, \sigma^2 \sum_{i=1}^n k_i^2)$.

For (3), I'm not sure where to start. Using the definition of $\mathcal{L}(\beta; y)$ above, I simplified to be $$\mathcal{L}(\beta; y) = \exp \left[ \frac{1}{2\sigma^2} \left(2\beta \sum_{i=1}^n x_i y_i - \beta^2 \sum_{i=1}^nx_i^2 - \frac{(\sum_{i=1}^n x_i y_i)^2}{\sum_{i=1}^n x_i^2} \right) \right]$$ So $-2\ln \mathcal{L}(\beta; y)$ would be $$ \frac{-1}{\sigma^2} \left(2\beta \sum_{i=1}^n x_i y_i - \beta^2 \sum_{i=1}^nx_i^2 - \frac{(\sum_{i=1}^n x_i y_i)^2}{\sum_{i=1}^n x_i^2} \right)$$ but I'm not seeing how this is a chi-squared distribution with 1 degree of freedom.

I realize this is a long post, but I'd appreciate any hints or clarifications on either subproblem.

Answer 1

For (2), re-instate the expression for the $k_i$'s and see what this gives you (there was a typo in what you wrote, now corrected).

For (3) a strategy here would be to show that

$$-2\ln \mathcal{L}(\beta; y) = -2 \ln f(y ; \beta) +2\ln f(y ; \hat{\beta})$$

is the difference of two chi-squares, one with one degree of freedom less than the other (chi-squares are closed under addition and subtraction).

We have that

$$-2\ln \mathcal{L}(\beta; y) = \sum_{i=1}^n\left(\frac {y_i -\beta x_i}{\sigma}\right)^2 - \sum_{i=1}^n\left(\frac {y_i -\hat \beta x_i}{\sigma}\right)^2$$

Given the assumptions the first sum is a sum of $n$ independent standard normals, and so a chi-square with $n$ degrees of freedom. So what you have to show is that the second sum is a chi-square with $n-1$ degrees of freedom. Intuitively, we should expect so, because what you have is again a sum of standardized normal random variables but by their estimated mean rather than the true one (so "you lose one degree of freedom"). This can be viewed as an application of the general form of Cochran's Theorem. Can you take it from here?

Answer 2

You need to simplify (2) much more.

Recall that a linear combination of independent normals is itself normal; namely if $Y_i \sim \operatorname{Normal}(\mu_i, \sigma_i^2)$, then $$\sum c_i Y_i \sim \operatorname{Normal}\left( \sum c_i \mu_i, \sum c_i \sigma_i^2 \right).$$ So if our realizations $y_i$ are drawn from $Y_i \sim \operatorname{Normal}(\beta x_i, \sigma^2)$, then the choice $c_i = x_i$, $\mu_i = \beta x_i$, $\sigma_i = \sigma$ immediately gives $$\sum x_i y_i \sim \operatorname{Normal}\left( \beta \sum x_i^2, \sigma^2 \sum x_i^2 \right).$$ Therefore $$\hat \beta = \frac{\sum x_i y_i}{\sum x_i^2} \sim \operatorname{Normal}\left( \beta, \frac{\sigma^2}{\sum x_i^2} \right).$$

To motivate part (3), write $$\mathcal L(\beta \mid \boldsymbol y) \propto \prod f(y_i \mid \beta) \propto \exp \sum -\frac{(y_i - \beta_i x_i)^2}{2\sigma^2}$$ hence $$-2 \ell(\beta \mid \boldsymbol y) \propto \sum \left(\frac{y_i - \beta_i x_i}{\sigma}\right)^2.$$ What is the distribution of this? Since $y_i \sim \operatorname{Normal}(\beta_i x_i, \sigma^2)$, it follows that $$\frac{y_i - \beta_i x_i}{\sigma} \sim \operatorname{Normal}(0,1),$$ and the sum of IID standard normals is chi-squared, but here we have the wrong degrees of freedom because we disregarded a lot of details in the interest of getting some understanding. Based on the above, I think you should try your hand at doing the calculation while incorporating $f(\boldsymbol y, \hat \beta)$.