I'm reading an explanation of maximum likelihood for a linear regression. The author has this to say where he derives the log likelihood as follows:

"As the data points are independent, we can write the joint probability distribution of $y, \theta, \sigma$ as:

$p(y \vert X, \theta, \sigma) = \prod_{i=1}^{n} p(y_i \vert x_i, \theta, \sigma)\\ p(y \vert X, \theta, \sigma) = \prod_{i=1}^{n} (2\pi\sigma^2)^{-1/2} e^{-\frac{1}{2\sigma^2}(y_i - x_i^T\theta)^2}$

rewriting in vector form,

$p(y \vert X, \theta, \sigma) = (2\pi\sigma^2)^{-n/2} e^{-\frac{1}{2\sigma^2}(y - X\theta)^T(y - X\theta)}$

Log likelihood,

$l(\theta) = -\frac{n}{2}log(2\pi\sigma^2) -\frac{1}{2\sigma^2}(Y-X\theta)^T(Y-X\theta)$"

What I'd like to know is, is the equation that he writes in vector form the likelihood i.e. this part: $p(y \vert X, \theta, \sigma) = (2\pi\sigma^2)^{-n/2} e^{-\frac{1}{2\sigma^2}(y - X\theta)^T(y - X\theta)}$


The linear regression model with normal errors can be written as $Y = X\beta + \varepsilon$ where $\varepsilon \sim \mathcal N(\vec 0, \sigma^2 I_n)$. This means $Y | \{X, \beta, \sigma^2\} \sim \mathcal N(X\beta, \sigma^2 I)$ so, using the likelihood of the multivariate normal, we have $$ p(y | X, \beta, \sigma^2) = \frac{1}{(2\pi)^{n/2}\left(\det \sigma^2 I\right)^{\frac 12}} \exp\left(-\frac 12 (y - X\beta)^T (\sigma^2 I)^{-1}(y - X\beta)\right). $$ Because our covariance matrix is so nice this reduces to a much simpler form, namely $$ p(y | X, \beta, \sigma^2) = \frac{1}{(2\pi \sigma^2)^{n/2}} \exp\left(-\frac 1{2\sigma^2} (y - X\beta)^T(y - X\beta)\right) $$

which is indeed what that author has written. This shows that this is "the" likelihood and we can just go directly to this form without needing to first consider marginal densities and combine them via independence, although of course the two are equivalent.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.