2
$\begingroup$

We have $$\mathbb{E}[Y_i| X_i] = β_0 + β_1X_i$$

What would be the Method of Moments estimator and MLE for this model?

$\endgroup$
7
  • $\begingroup$ For the MLE, do you know how to take the partial derivative of log-likelihood in terms of $\beta_0, \beta_1$ and $\sigma^2 $? for the information matrix you can search the second derivative of log-likelihood, Fisher information matrix, Hessian to see what are they. $\endgroup$
    – Deep North
    Dec 13, 2020 at 11:41
  • $\begingroup$ Yes, I've already found the partial derivatives in terms of $\beta_0, \beta_1$, and $\sigma^2$. So for the information matrix just take the second derivative of the log likelihood? $\endgroup$ Dec 13, 2020 at 11:50
  • $\begingroup$ A few things: 1) typo: last derivative is with respect to $\sigma$, not $\beta_0$; 2) on the same line, $n$ is missing from the first term ($n/2$ remains as factor); 3) derivative of $2\sigma^{-2}$ is $-4\sigma^{-3}$, not $-2(\sigma^2)^2$. $\endgroup$
    – PaulG
    Dec 13, 2020 at 12:12
  • $\begingroup$ @PaulG thank you for pointing my typos, already fixed them. Besides that, is the solution correct? $\endgroup$ Dec 13, 2020 at 12:17
  • $\begingroup$ If you go through with the optimization and solve for $\beta_0$ and $\beta_1$ (for MLE) you should get the same estimator as for OLS. Then you'd know you did the right thing. $\endgroup$
    – PaulG
    Dec 13, 2020 at 12:57

1 Answer 1

2
$\begingroup$

Now, you have the score function:

$$U =\begin{pmatrix} \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \beta_0} \\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \beta_1}\\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \sigma^2}\\ \end{pmatrix}$$

You would need to compute the information matrix $I$ in this way computing the derivative of $U$:

$$I =\begin{pmatrix} \frac{\partial log(L(β_0, β_1, σ^2)}{\partial^2 \beta_0} &.&. \\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \beta_1\beta_0}&\frac{\partial log(L(β_0, β_1, σ^2)}{\partial^2 \beta_1}&.\\ \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \sigma^2\beta_0}& \frac{\partial log(L(β_0, β_1, σ^2)}{\partial \sigma^2\beta_1}&\frac{\partial log(L(β_0, β_1, σ^2)}{\partial^2 \sigma^2}\\ \end{pmatrix}$$

The dots are related to symmetry as upper and lower diagonal are equal. After some math, you will end up with this:

$$I=\begin{pmatrix} -\frac{n}{\sigma^2} &.&. \\ -\frac{\sum_{i=1}^{n}x_i}{\sigma^2}&-\frac{\sum_{i=1}^{n}x_i^2}{\sigma^2}&.\\ -\frac{2}{\sigma^3}\sum_{i=1}^{n}(Y_i − β_0 − β_1x_i)& -\frac{2}{\sigma^3}\sum_{i=1}^{n}x_i(Y_i-\beta_0-\beta_1x_i)&\frac{n}{\sigma^3}-\frac{2}{\sigma^{5}}\sum_{i=1}^{n}(Y_i − β_0 − β_1x_i)^2=0\\ \end{pmatrix}$$

Which is the information matrix.

$\endgroup$
4
  • $\begingroup$ Thank you for your help. Could I ask if the analysis would change if we were to drop the assumptions that $X_i$ is fixed in repeated samples, but the samples are still random? $\endgroup$ Dec 14, 2020 at 16:52
  • $\begingroup$ @MaybelineLee As it is a linar model, if Xi changes, the model matrix will change and estimators will be different althought the method of derivatives will still be the same. $\endgroup$
    – Duck
    Dec 14, 2020 at 17:34
  • $\begingroup$ By information matrix, do you mean Fisher information matrix? $\endgroup$
    – develarist
    Dec 16, 2020 at 23:27
  • $\begingroup$ @develarist yes $\endgroup$ Dec 17, 2020 at 6:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.