Given the following simple linear regression model:

$$y = ax + \epsilon$$

where the variance is normally distributed, I'd like to learn the maximum likelihood estimates for $a$ (which is a scalar) and $\sigma^2$. I'm confident in my estimate for $\sigma^2$ but not $a$.

First I derived the log likelihood. I obtained:

$$\frac{-N}2log(2\pi) - \frac{-N}2log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i-ax_i)^2$$

After calculating the derivative of the log likelihood with respect to $a$, I obtain:

$$\frac{1}{\sigma^2}\sum_{i=1}^n (y_i-ax_i)x_i = 0$$

Assuming this is right (it's been a long time since I took calc) then wouldn't the estimate for $a$ be the following?

$$\frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n x_i}$$

But this doesn't seem to match any of the other solutions I've seen on CV or elsewhere for this estimate (i.e. http://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/05/lecture-05.pdf)

I apologize if I've made a stupid math error here. Any tips would be much appreciated. Thanks!


Your last step is wrong.

$$\frac{1}{\sigma^2}\sum_{i=1}^n (y_i-ax_i)x_i = 0 \\ \sum_{i=1}^n (y_i-ax_i)x_i = 0 \\ \sum_{i=1}^nx_iy_i-\sum_{i=1}^nax_i^2=0\\a=\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}$$

This should match all other solution suppose you have no intercept

| cite | improve this answer | |
  • $\begingroup$ Deep North, thanks very much for catching my careless mistake. This indeed matches other solutions! $\endgroup$ – sqlck Sep 3 '17 at 0:10

You want to start with:

$\frac{\sum_{i=1}^n (y_i - ax_i)x_i}{\sigma^2} = 0,$ then you can eliminate the $\sigma^2$ term to get:

$\sum_{i=1}^n (y_ix_i - ax_i^2).$

Then solving for $a$ gives $\hat{a} = \frac{\sum_{i=1}^n y_ix_i}{\sum_{i=1}^n x_i^2}$. Next, check that the second derivates is strictly greater than $0$ to argue that $\hat{a}$ gives you a maximum.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.