# Method of Moment, MLE and Information matrix

We have $$\mathbb{E}[Y_i| X_i] = β_0 + β_1X_i$$

What would be the Method of Moments estimator and MLE for this model?

• For the MLE, do you know how to take the partial derivative of log-likelihood in terms of $\beta_0, \beta_1$ and $\sigma^2$? for the information matrix you can search the second derivative of log-likelihood, Fisher information matrix, Hessian to see what are they. Dec 13, 2020 at 11:41
• Yes, I've already found the partial derivatives in terms of $\beta_0, \beta_1$, and $\sigma^2$. So for the information matrix just take the second derivative of the log likelihood? Dec 13, 2020 at 11:50
• A few things: 1) typo: last derivative is with respect to $\sigma$, not $\beta_0$; 2) on the same line, $n$ is missing from the first term ($n/2$ remains as factor); 3) derivative of $2\sigma^{-2}$ is $-4\sigma^{-3}$, not $-2(\sigma^2)^2$. Dec 13, 2020 at 12:12
• @PaulG thank you for pointing my typos, already fixed them. Besides that, is the solution correct? Dec 13, 2020 at 12:17
• If you go through with the optimization and solve for $\beta_0$ and $\beta_1$ (for MLE) you should get the same estimator as for OLS. Then you'd know you did the right thing. Dec 13, 2020 at 12:57

Now, you have the score function:

$$U =\begin{pmatrix} \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \beta_0} \\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \beta_1}\\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \sigma^2}\\ \end{pmatrix}$$

You would need to compute the information matrix $$I$$ in this way computing the derivative of $$U$$:

$$I =\begin{pmatrix} \frac{\partial log(L(β_0, β_1, σ^2)}{\partial^2 \beta_0} &.&. \\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \beta_1\beta_0}&\frac{\partial log(L(β_0, β_1, σ^2)}{\partial^2 \beta_1}&.\\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \sigma^2\beta_0}& \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \sigma^2\beta_1}&\frac{\partial log(L(β_0, β_1, σ^2)}{\partial^2 \sigma^2}\\ \end{pmatrix}$$

The dots are related to symmetry as upper and lower diagonal are equal. After some math, you will end up with this:

$$I=\begin{pmatrix} -\frac{n}{\sigma^2} &.&. \\ -\frac{\sum_{i=1}^{n}x_i}{\sigma^2}&-\frac{\sum_{i=1}^{n}x_i^2}{\sigma^2}&.\\ -\frac{2}{\sigma^3}\sum_{i=1}^{n}(Y_i − β_0 − β_1x_i)& -\frac{2}{\sigma^3}\sum_{i=1}^{n}x_i(Y_i-\beta_0-\beta_1x_i)&\frac{n}{\sigma^3}-\frac{2}{\sigma^{5}}\sum_{i=1}^{n}(Y_i − β_0 − β_1x_i)^2=0\\ \end{pmatrix}$$

Which is the information matrix.

• Thank you for your help. Could I ask if the analysis would change if we were to drop the assumptions that $X_i$ is fixed in repeated samples, but the samples are still random? Dec 14, 2020 at 16:52
• @MaybelineLee As it is a linar model, if Xi changes, the model matrix will change and estimators will be different althought the method of derivatives will still be the same.
– Duck
Dec 14, 2020 at 17:34
• By information matrix, do you mean Fisher information matrix? Dec 16, 2020 at 23:27
• @develarist yes Dec 17, 2020 at 6:10