Maximum Likelihood, Simple Linear Regression

Question

Given the following simple linear regression model:

$$y = ax + \epsilon$$

where the variance is normally distributed, I'd like to learn the maximum likelihood estimates for $a$ (which is a scalar) and $\sigma^2$. I'm confident in my estimate for $\sigma^2$ but not $a$.

First I derived the log likelihood. I obtained:

$$\frac{-N}2log(2\pi) - \frac{-N}2log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i-ax_i)^2$$

After calculating the derivative of the log likelihood with respect to $a$, I obtain:

$$\frac{1}{\sigma^2}\sum_{i=1}^n (y_i-ax_i)x_i = 0$$

Assuming this is right (it's been a long time since I took calc) then wouldn't the estimate for $a$ be the following?

$$\frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n x_i}$$

But this doesn't seem to match any of the other solutions I've seen on CV or elsewhere for this estimate (i.e. http://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/05/lecture-05.pdf)

I apologize if I've made a stupid math error here. Any tips would be much appreciated. Thanks!

Answer 1

Your last step is wrong.

$$\frac{1}{\sigma^2}\sum_{i=1}^n (y_i-ax_i)x_i = 0 \\ \sum_{i=1}^n (y_i-ax_i)x_i = 0 \\ \sum_{i=1}^nx_iy_i-\sum_{i=1}^nax_i^2=0\\a=\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}$$

This should match all other solution suppose you have no intercept

Answer 2

You want to start with:

$\frac{\sum_{i=1}^n (y_i - ax_i)x_i}{\sigma^2} = 0,$ then you can eliminate the $\sigma^2$ term to get:

$\sum_{i=1}^n (y_ix_i - ax_i^2).$

Then solving for $a$ gives $\hat{a} = \frac{\sum_{i=1}^n y_ix_i}{\sum_{i=1}^n x_i^2}$. Next, check that the second derivates is strictly greater than $0$ to argue that $\hat{a}$ gives you a maximum.