In an assignment I need to show that for linear regression, with the noise i.i.d. Gaussian distributed $\epsilon_i \sim N(0,\sigma^2)$, that finding the Maximum Likelihood Estimate (MLE) is equivalent to finding $\hat{y}$, which minimizes the sum of the least square errors to the $x$'s.

From Wikipedia's page on linear regression I saw a section stating that the MLE is identical to the OLS (Ordinary Least Squares) estimate when the noise has the above-mentioned distribution. Could this help solve the problem? I would greatly appreciate a step-by-step guide to solving this problem.

  • $\begingroup$ If it's an assignment please add the self-study tag. $\endgroup$ – Christoph Hanck Apr 2 '15 at 4:34
  • $\begingroup$ I edited your question - please check if it is ok. I changed $\epsilon_i - N(0,\sigma^2)$ to $\epsilon_i \sim N(0,\sigma^2)$ since I imagine it was a mistake but if not feel free to revert my edit. $\endgroup$ – Tim Apr 2 '15 at 6:18

Let $\mu_i = \alpha + x_i^T\beta$ where $x_i$ is a p-dimensional vector of observed covariates, and $\beta$ is a p-dimensional vector of regression coefficients. Then let $y_i \overset{ind}{\sim} N(\mu_i,\sigma^2)$. Write the joint distribution of the $y_i$'s and find the log-likelihood. You will see that the objective function for $\alpha$ and $\beta$ is the sum of squares $\sum_{i=1}^n (y_i - \mu_i)^2$ as it is for a least squares problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.