5
$\begingroup$

I keep coming across two different kinds of optimization:

  1. Cases where you maximize the likelihood of the data directly (for example CRF learning, or EM).
  2. Cases where you minimize some cost function (for example, fitting least squares)

I also have noticed that people use gradient methods for solving each of these two kinds of problems.

For maximization, the gradient update rule looks like this. The intuition is you want to maximize so you climb the hill of the curvature in the direction of the gradient.

$$\lambda_{i+1} = \lambda_i + \frac{\partial f(x)}{\partial \lambda_i}$$

For minimization, you want to minimize the cost function, so you subtract the gradient to roll down the hill of the curvature.

$$\lambda_{i+1} = \lambda_i - \frac{\partial f(x)}{\partial \lambda_i}$$

It also seems like some optimization packages ask you to flip the sign of a maximization problem to get a minimization problem instead. Example:

Note that since minimize only minimizes functions, the sign parameter is introduced to multiply the objective function (and its derivative) by -1 in order to perform a maximization.

  1. Is minimization more canonical?
  2. Am I getting this right? That is, is my description of the machine learning landscape correct?
  3. How do I know when I should minimize a cost function or maximize a likelihood? (Or a log likelihood).

My first thought was that maximizing log likelihood is for unsupervised learning (where you can't generate a cost function, because there are no labels) -- but CRF learning maximizes log likelihood directly too.

Improve this question
$\endgroup$
5
  • $\begingroup$ I don't understand where you're stuck. Question (1) asks if ML uses gradient-based optimization? But you've provided examples that show that ML does, in fact, use gradient optimization to solve some problems. And question (2) asks how one knows whether to maximize or minimize functions. But objective functions say "min" or "max" as part of the expression, so I doubt you're confused about that. $\endgroup$
    – Sycorax ♦
    Commented Nov 20, 2015 at 14:55
  • 1
    $\begingroup$ You already know a lot. Two observations. Take linear regression. Minimizing the squared error turns out to be equivalent to maximizing the likelihood. Loosely one could say that minimizing the squared error is an intuitive method, and maximizing the likelihood a more formal approach that allows for proofs. The outcomes can overlap. Second minimizing or maximizing is often AFAIK arbitrary. Minimizing the negative is the same as maximizing the positive. There are a lot of routines that are written in the minimization mode: this is sort of coincidence. $\endgroup$
    – spdrnl
    Commented Nov 20, 2015 at 15:11
  • $\begingroup$ @spdrnl, why not turn that into an official answer? $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 20, 2015 at 16:02
  • $\begingroup$ @gung Thx ;-) Done. $\endgroup$
    – spdrnl
    Commented Nov 20, 2015 at 16:07
  • $\begingroup$ It might be helpful in noting a sort of duality between loss functions and likelihoods. Given a loss function $L(y, \theta)$ I can create a pseudo-likelihood $\exp(-L(y, \theta))$. In this setting, maximizing the likelihood and minimizing the loss are the same thing. Often, as in linear regression, this pseudo-likelihood corresponds to a genuine likelihood; in general it won't, but there is a literature concerning the use of such pseudo-likelihoods for inference purposes. $\endgroup$
    – guy
    Commented Nov 20, 2015 at 16:33

1 Answer 1

Reset to default
7
$\begingroup$

You already know a lot. Two observations.

Take linear regression. Minimizing the squared error turns out to be equivalent to maximizing the likelihood. Loosely one could say that minimizing the squared error is an intuitive method, and maximizing the likelihood a more formal approach that allows for proofs using properties of for example the normal distribution. The outcomes can overlap.

Second minimizing or maximizing is often AFAIK arbitrary. Minimizing the negative is the same as maximizing the positive. There are a lot of routines that are written in the minimization mode: this is sort of coincidence. For reasons of parsimony/readability this has become standard.

Improve this answer
$\endgroup$
3
  • $\begingroup$ +1, FWIW I was under the impression that although maximizing & minimizing are mathematically equivalent (in the sense of having the same optimum), there is some computer science reason why it is better (easier, faster, more stable) to minimize. That's just something I've heard, it may not be true. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 20, 2015 at 16:19
  • $\begingroup$ would agree with spdrnl that minimisation (vs max) is just a convention $\endgroup$
    – seanv507
    Commented Nov 21, 2015 at 22:43
  • $\begingroup$ @gung-ReinstateMonica: Why: the numerical operations are exactly the same, modulo a minus sign. I think the only computer science reason is software availability and convention $\endgroup$
    – kjetil b halvorsen ♦
    Commented Dec 30, 2022 at 14:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.