3
$\begingroup$

I keep coming across two different kinds of optimization:

  1. Cases where you maximize the likelihood of the data directly (for example CRF learning, or EM).
  2. Cases where you minimize some cost function (for example, fitting least squares)

I also have noticed that people use gradient methods for solving each of these two kinds of problems.

For maximization, the gradient update rule looks like this. The intuition is you want to maximize so you climb the hill of the curvature in the direction of the gradient.

$$\lambda_{i+1} = \lambda_i + \frac{\partial f(x)}{\partial \lambda_i}$$

For minimization, you want to minimize the cost function, so you subtract the gradient to roll down the hill of the curvature.

$$\lambda_{i+1} = \lambda_i - \frac{\partial f(x)}{\partial \lambda_i}$$

It also seems like some optimization packages ask you to flip the sign of a maximization problem to get a minimization problem instead. Example:

Note that since minimize only minimizes functions, the sign parameter is introduced to multiply the objective function (and its derivative) by -1 in order to perform a maximization.

  1. Is minimization more canonical?
  2. Am I getting this right? That is, is my description of the machine learning landscape correct?
  3. How do I know when I should minimize a cost function or maximize a likelihood? (Or a log likelihood).

My first thought was that maximizing log likelihood is for unsupervised learning (where you can't generate a cost function, because there are no labels) -- but CRF learning maximizes log likelihood directly too.

$\endgroup$
  • $\begingroup$ I don't understand where you're stuck. Question (1) asks if ML uses gradient-based optimization? But you've provided examples that show that ML does, in fact, use gradient optimization to solve some problems. And question (2) asks how one knows whether to maximize or minimize functions. But objective functions say "min" or "max" as part of the expression, so I doubt you're confused about that. $\endgroup$ – Sycorax Nov 20 '15 at 14:55
  • 1
    $\begingroup$ You already know a lot. Two observations. Take linear regression. Minimizing the squared error turns out to be equivalent to maximizing the likelihood. Loosely one could say that minimizing the squared error is an intuitive method, and maximizing the likelihood a more formal approach that allows for proofs. The outcomes can overlap. Second minimizing or maximizing is often AFAIK arbitrary. Minimizing the negative is the same as maximizing the positive. There are a lot of routines that are written in the minimization mode: this is sort of coincidence. $\endgroup$ – spdrnl Nov 20 '15 at 15:11
  • $\begingroup$ @spdrnl, why not turn that into an official answer? $\endgroup$ – gung Nov 20 '15 at 16:02
  • $\begingroup$ @gung Thx ;-) Done. $\endgroup$ – spdrnl Nov 20 '15 at 16:07
  • $\begingroup$ It might be helpful in noting a sort of duality between loss functions and likelihoods. Given a loss function $L(y, \theta)$ I can create a pseudo-likelihood $\exp(-L(y, \theta))$. In this setting, maximizing the likelihood and minimizing the loss are the same thing. Often, as in linear regression, this pseudo-likelihood corresponds to a genuine likelihood; in general it won't, but there is a literature concerning the use of such pseudo-likelihoods for inference purposes. $\endgroup$ – guy Nov 20 '15 at 16:33
6
$\begingroup$

You already know a lot. Two observations.

Take linear regression. Minimizing the squared error turns out to be equivalent to maximizing the likelihood. Loosely one could say that minimizing the squared error is an intuitive method, and maximizing the likelihood a more formal approach that allows for proofs using properties of for example the normal distribution. The outcomes can overlap.

Second minimizing or maximizing is often AFAIK arbitrary. Minimizing the negative is the same as maximizing the positive. There are a lot of routines that are written in the minimization mode: this is sort of coincidence. For reasons of parsimony/readability this has become standard.

$\endgroup$
  • $\begingroup$ +1, FWIW I was under the impression that although maximizing & minimizing are mathematically equivalent (in the sense of having the same optimum), there is some computer science reason why it is better (easier, faster, more stable) to minimize. That's just something I've heard, it may not be true. $\endgroup$ – gung Nov 20 '15 at 16:19
  • $\begingroup$ would agree with spdrnl that minimisation (vs max) is just a convention $\endgroup$ – seanv507 Nov 21 '15 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.