25
$\begingroup$

I am new to Machine Learning, and am trying to learn it on my own. Recently I was reading through some lecture notes and had a basic question.

Slide 13 says that "Least Square Estimate is same as Maximum Likelihood Estimate under a Gaussian model". It seems like it is something simple, but I am unable to see this. Can someone please explain what is going on here? I am interested in seeing the Math.

I will later try to see the probabilistic viewpoint of Ridge and Lasso regression also, so if there are any suggestions that will help me, that will be much appreciated also.

$\endgroup$
  • 4
    $\begingroup$ The objective function at the bottom of p. 13 is just a constant multiple ($n$) of the objective function at the bottom of p. 10. MLE minimizes the former while least squares minimizes the latter, QED. $\endgroup$ – whuber Jul 1 '11 at 21:37
  • $\begingroup$ @whuber : Thank you for your answer. Well what I was wanting to know is how is it that MLE is doing the minimization. $\endgroup$ – Andy Jul 1 '11 at 21:45
  • $\begingroup$ Do you mean the mechanics or conceptually? $\endgroup$ – whuber Jul 1 '11 at 21:47
  • $\begingroup$ @whuber: Both ! If I could see that Math, that will help also. $\endgroup$ – Andy Jul 1 '11 at 21:48
  • 1
    $\begingroup$ The link is broken; the lack of a full reference and more context for the quote makes it difficult to just remove the reference or locate an alternative source for it. Is slide 13 of this link sufficient? --- cs.cmu.edu/~epxing/Class/10701-10s/recitation/recitation3.pdf $\endgroup$ – Glen_b Oct 2 '17 at 4:38
28
$\begingroup$

In the model

$ Y = X \beta + \epsilon $

where $\epsilon \sim N(0,\sigma^{2})$, the loglikelihood of $Y|X$ for a sample of $n$ subjects is (up to a additive constant)

$$ \frac{-n}{2} \log(\sigma^{2}) - \frac{1}{2 \sigma^{2}} \sum_{i=1}^{n} (y_{i}-x_{i} \beta)^{2} $$

viewed as a function of only $\beta$, the maximizer is exactly that which minimizes

$$ \sum_{i=1}^{n} (y_{i}-x_{i} \beta)^{2} $$

does this make the equivalence clear?

$\endgroup$
  • $\begingroup$ This is precisely what is in the slides referred to in the O.P. $\endgroup$ – whuber Jul 2 '11 at 18:11
  • 3
    $\begingroup$ Yes I see that but they do not actually write the Gaussian log-likelihood on page 13 which, after doing so, makes it's obvious that its argmax is the same as the OLS criteria's argmin, so I thought this was a worthwhile addition. $\endgroup$ – Macro Jul 2 '11 at 20:42
  • $\begingroup$ good point: the slide is a little sketchy with the details. $\endgroup$ – whuber Jul 2 '11 at 21:01
  • 7
    $\begingroup$ You've learned that, if you know the errors are normally distributed around the regression line then the least squares estimator is "optimal" in some sense, other than arbitrarily decreeing that "least squares" is best. Regarding ridge regression, this solution is equivalent (if you are a bayesian) to the least squares estimator when a Gaussian prior is placed on the $\beta$'s. In a frequentist world it is equivalent to $L_{2}$ penalized least squares. Logistic regression coefficients are not the solution to a least squares problem, so that would not be analogous. $\endgroup$ – Macro Jul 3 '11 at 1:08
  • 1
    $\begingroup$ The additive constant is n/2 log(2 *pi) $\endgroup$ – SmallChess Jan 27 '17 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.