1
$\begingroup$

Given the following simple linear regression model:

$$y = ax + \epsilon$$

where the variance is normally distributed, I'd like to learn the maximum likelihood estimates for $a$ (which is a scalar) and $\sigma^2$. I'm confident in my estimate for $\sigma^2$ but not $a$.

First I derived the log likelihood. I obtained:

$$\frac{-N}2log(2\pi) - \frac{-N}2log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i-ax_i)^2$$

After calculating the derivative of the log likelihood with respect to $a$, I obtain:

$$\frac{1}{\sigma^2}\sum_{i=1}^n (y_i-ax_i)x_i = 0$$

Assuming this is right (it's been a long time since I took calc) then wouldn't the estimate for $a$ be the following?

$$\frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n x_i}$$

But this doesn't seem to match any of the other solutions I've seen on CV or elsewhere for this estimate (i.e. http://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/05/lecture-05.pdf)

I apologize if I've made a stupid math error here. Any tips would be much appreciated. Thanks!

$\endgroup$

2 Answers 2

5
$\begingroup$

Your last step is wrong.

$$\frac{1}{\sigma^2}\sum_{i=1}^n (y_i-ax_i)x_i = 0 \\ \sum_{i=1}^n (y_i-ax_i)x_i = 0 \\ \sum_{i=1}^nx_iy_i-\sum_{i=1}^nax_i^2=0\\a=\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}$$

This should match all other solution suppose you have no intercept

$\endgroup$
1
  • $\begingroup$ Deep North, thanks very much for catching my careless mistake. This indeed matches other solutions! $\endgroup$
    – sqlck
    Sep 3, 2017 at 0:10
0
$\begingroup$

You want to start with:

$\frac{\sum_{i=1}^n (y_i - ax_i)x_i}{\sigma^2} = 0,$ then you can eliminate the $\sigma^2$ term to get:

$\sum_{i=1}^n (y_ix_i - ax_i^2).$

Then solving for $a$ gives $\hat{a} = \frac{\sum_{i=1}^n y_ix_i}{\sum_{i=1}^n x_i^2}$. Next, check that the second derivates is strictly greater than $0$ to argue that $\hat{a}$ gives you a maximum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.