38
$\begingroup$

I am new to Machine Learning, and am trying to learn it on my own. Recently I was reading through some lecture notes and had a basic question.

Slide 13 says that "Least Square Estimate is same as Maximum Likelihood Estimate under a Gaussian model". It seems like it is something simple, but I am unable to see this. Can someone please explain what is going on here? I am interested in seeing the Math.

I will later try to see the probabilistic viewpoint of Ridge and Lasso regression also, so if there are any suggestions that will help me, that will be much appreciated also.

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ The objective function at the bottom of p. 13 is just a constant multiple ($n$) of the objective function at the bottom of p. 10. MLE minimizes the former while least squares minimizes the latter, QED. $\endgroup$
    – whuber ♦
    Commented Jul 1, 2011 at 21:37
  • $\begingroup$ @whuber : Thank you for your answer. Well what I was wanting to know is how is it that MLE is doing the minimization. $\endgroup$
    – Andy
    Commented Jul 1, 2011 at 21:45
  • $\begingroup$ Do you mean the mechanics or conceptually? $\endgroup$
    – whuber ♦
    Commented Jul 1, 2011 at 21:47
  • $\begingroup$ @whuber: Both ! If I could see that Math, that will help also. $\endgroup$
    – Andy
    Commented Jul 1, 2011 at 21:48
  • 1
    $\begingroup$ The link is broken; the lack of a full reference and more context for the quote makes it difficult to just remove the reference or locate an alternative source for it. Is slide 13 of this link sufficient? --- cs.cmu.edu/~epxing/Class/10701-10s/recitation/recitation3.pdf $\endgroup$
    – Glen_b
    Commented Oct 2, 2017 at 4:38

1 Answer 1

Reset to default
38
$\begingroup$

In the model

$ Y = X \beta + \epsilon $

where $\epsilon \sim N(0,\sigma^{2})$, the loglikelihood of $Y|X$ for a sample of $n$ subjects is (up to a additive constant)

$$ \frac{-n}{2} \log(\sigma^{2}) - \frac{1}{2 \sigma^{2}} \sum_{i=1}^{n} (y_{i}-x_{i} \beta)^{2} $$

viewed as a function of only $\beta$, the maximizer is exactly that which minimizes

$$ \sum_{i=1}^{n} (y_{i}-x_{i} \beta)^{2} $$

does this make the equivalence clear?

Improve this answer
$\endgroup$
7
  • $\begingroup$ This is precisely what is in the slides referred to in the O.P. $\endgroup$
    – whuber ♦
    Commented Jul 2, 2011 at 18:11
  • 4
    $\begingroup$ Yes I see that but they do not actually write the Gaussian log-likelihood on page 13 which, after doing so, makes it's obvious that its argmax is the same as the OLS criteria's argmin, so I thought this was a worthwhile addition. $\endgroup$
    – Macro
    Commented Jul 2, 2011 at 20:42
  • $\begingroup$ good point: the slide is a little sketchy with the details. $\endgroup$
    – whuber ♦
    Commented Jul 2, 2011 at 21:01
  • 7
    $\begingroup$ You've learned that, if you know the errors are normally distributed around the regression line then the least squares estimator is "optimal" in some sense, other than arbitrarily decreeing that "least squares" is best. Regarding ridge regression, this solution is equivalent (if you are a bayesian) to the least squares estimator when a Gaussian prior is placed on the $\beta$'s. In a frequentist world it is equivalent to $L_{2}$ penalized least squares. Logistic regression coefficients are not the solution to a least squares problem, so that would not be analogous. $\endgroup$
    – Macro
    Commented Jul 3, 2011 at 1:08
  • 1
    $\begingroup$ The additive constant is n/2 log(2 *pi) $\endgroup$
    – SmallChess
    Commented Jan 27, 2017 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.