I am beginner in machine learning and I am currently trying to find the motivation for gradient descent method. I am confused why we want to employ gradient descent method for linear regression? I see the cost function the same as the OLS function, and gradient descent method here actually takes more effort than simply getting the derivatives equal zero. Then why we always try to use gradient descent here? I am when the model gets more complicated , and also when we make more assumptions on the prior distribution of the theta(parameters). The optimization problem will become much more complicated. Then will gradient descent method still survive in terms of this? And OLS/MLE method will not be able to predict the parameters? I see OLS as minimize the cost, and the MLE method as maximize the prob, which is in essence the same.(reference http://www.cs.ubc.ca/~nando/540-2013/lectures/l3.pdf) Should I think gradient descent method as a improvement from the OLS method, while the E-M method(maximize the expected likelihood) as a imporvement from the MLE method. Thanks in advance!

  • 1
    $\begingroup$ That's like asking "motivation for driving to get to the cinema over being at the cinema". You're confusing the journey with the destination. OLS is a destination. MLE is a destination. Gradient descent is one way to get to a destination. $\endgroup$ – Glen_b Mar 26 '15 at 5:13
  • $\begingroup$ @Glen_b Thanks!I will edit the problem, I mistook the lagrange method with OLS.You are right, I confused the journey with destination. Just to make sure, so OLS means minimize the cost, and MLE means maximize the likelihood? Does OLS and MLE always the same for any models? So that means it is the same destination(estimate the parameters), but different direction(OLS and MLE) $\endgroup$ – DQ_happy Mar 26 '15 at 5:32
  • $\begingroup$ OLS does not mean 'minimize the cost' unless the cost is given by the sums of squared residuals. $\endgroup$ – Glen_b Mar 26 '15 at 5:52
  • $\begingroup$ OLS is only the same as MLE at the normal (assuming constant variance). OLS is MLE for the normal, but not for anything else. When you have y|x being normally distributed OLS and ML are the same destination, but otherwise they're different destinations. However, OLS is often useful even when it isn't ML (it's still BLUE, under the required conditions). $\endgroup$ – Glen_b Mar 26 '15 at 5:58

For ordinary linear regression, maximum likelihood and least squares are the same, i.e., give the same answer (the maximum likelihood solution is the least squares solution, if you derive the so called ``normal equations'' you'll see this, also see the book The Elements of Statistical Learning which discusses this).

But this is separate from how you find that solution. Gradient descent is only one method to find the solution, and it's actually quite a bad one at that (slow to converge). For example, Newton's method is much better for OLS (using various numerical algorithms to avoid inverting the Hessian directly).

But you are right in the sense that for very big problems, gradient descent becomes more useful because 2nd order methods like Newton's method can be computationally very expensive (again, there are approximations to that too).

I don't think EM is relevant for OLS, it can be useful for optimizing non-convex problems (OLS is convex).

  • $\begingroup$ Thanks for your reply!So gradient descent is just a method to find solution for OLS? Also I still have a problem. Is MLE and OLS always the same? I can understand they are the same for ordinary linear regression, but as for more complicated problems like neural network?GMM?LDA?..also I am wondering for the bayesian inference, MLE can somehow evolve to E-M, but does OLS still work? Many thank:D $\endgroup$ – DQ_happy Mar 26 '15 at 3:15
  • $\begingroup$ Ah I do not whether it is still called OLS when the model is not ordinary linear regression, or maybe I shall say: does minimize cost and maximize likelihood always the same for all the models? so that means we can say we have two direction to solve the parameters estimation problem? $\endgroup$ – DQ_happy Mar 26 '15 at 3:21
  • $\begingroup$ Not the same for all models: each model has its own loss function to be minimized (equivalent to the negative log likelihood for probabilistic models). Least squares is just the solution to one of these models (the Gaussian likelihood with known variance), it doesn't apply to all of them, for example, least squares does not minimize the logistic regression (binomial) negative log likelihood, so it's not useful there (although least squares is useful in practice here but that's beyond the scope of this comment). $\endgroup$ – purple51 Mar 26 '15 at 5:49
  • $\begingroup$ Thanks for the reply! I misunderstood OLS previously. But I still have a problem :does minimize cost(different model has different cost functions) and maximize likelihood always the same for all the models? $\endgroup$ – DQ_happy Mar 26 '15 at 5:54
  • $\begingroup$ Not all models have a likelihood. A support vector machine is not a probabilistic model, does not have a likelihood but has a cost function (=hinge loss). Example of models that do have likelihoods are linear regression, logistic regression, Poisson regression, Cox regression, etc. $\endgroup$ – purple51 Mar 26 '15 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.