Why typically minimizing a cost instead of maximizing a reward?

Question

I understand that, for example, maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood. It is indeed a simple change, but still an extra step taken (it seems) for the unique purpose of designing a loss function that will be minimized instead of maximized.

I wonder why this has become the standard in Machine Learning?

• Is there any numerical consideration that favors function minimization instead of maximization?
• Why has gradient descent become such a universal standard? (I have never seen a Deep Learning paper in which they use gradient ascent to directly maximize the likelihood)

Disclaimer :

I came across many similar questions, but none of which that have been truly answered. People typically just explain how both approaches are equivalent, or explain why we use the logarithm for numerical stability, but without explaining why minimization is favored over maximization. (See those two questions : 1, 2)

Answer 1

Minimising $f(x)$ is entirely equivalent to maximising $-f(x)$, in every aspect: result, numerical precision, computational complexity... everything.

Historically, the convention might have been established because of the "least squares" in linear regression (but don't take my word for it).

If it were the other way round, you'd be asking why we don't minimise some cost function...

Answer 2

You tagged this question with the tag "Maximum Likelihood". In maximum likelihood estimation you explicitly maximize an objective function (namely the likelihood).

It just so happens that for an observation that we assume to be drawn from a Gaussian random variable, the likelihood function usually takes a nice form after you take a logarithm. Then there is usually a leading negation, encouraging the entrepreneurial optimizer to switch away from maximizing the objective to minimizing the negative of objective, or roughly the "cost".

For discrete maximum likelihood estimation the "cost" also has another meaningful name since it takes the same form as the euclidean distance in the observation space. (Note that this notion of distance is always there whether or not you're doing discrete parameter estimation, but it's a little less obvious than the discrete ML estimate which just boils down to picking the nearest valid point). Since there's no such thing as negative distance there is a seemingly strong preference for minimizing the cost and not maximizing the objective in these cases.

You should feel comfortable swapping back and forth between minimizing a cost and maximizing an objective. There is a real reason that ML and MAP estimates specifically choose to maximize an objective function (pdf's are purely positive, and the highest values are quite interesting spots [the mode]), but the practical realization of an estimator is going to be several mathematical manipulations away from the textbook definition.

Answer 3

It's my understanding that the only reason for this distinction is that in numerical analysis, it's the standard to talk about convex optimization rather than concave optimization, even though they are really the same procedures. For example, if you do a google scholar search for "concave optimization", you get about 300,000 hits, but "convex optimization" gets about 2,000,000.

Because convex optimization is talked about more in the numerical analysis literature, this nomenclature is followed in the machine learning community.

As you state, the differences are trivial, so the reason for the distinction is trivial.

Answer 4

History.

A lot of this connects back to estimation in statistics.

For example Gauss. He wanted to estimate the position of an asteroid that was obscured by the sun. He had the idea to minimize the squared error and got much better predictions than his colleagues. When estimating the position of an asteroid, what would be the "gain"? The error cost, however, is easy to see: how far is the asteroid from the expected position.

Answer 5

I suspect it may also be because quite a lot of optimisation algorithms are developed by people working on Operations Research who I think have historically posed problems in terms of minimisation of losses. We minimise because it what the best software supports.

Answer 6

You can just as well maximize a function that is equal to -1 times your cost function. It just happens that it's usually more natural to define a function that increases in value the farther from the optimum we get than the reverse, and for that reason we usually try to minimize a function rather than maximize a function.

As an illustrative example of what I mean, see this table, where in the rightmost column you encounter $\operatorname{argmin}$ much more often than $\operatorname{argmax}$.