4
$\begingroup$

In a classification task, at a certain stage of the training process, we get a likelihood of sampling proper class Y for a particular data point X. For batch, we get many independent likelihoods.

Let's say we have likelihoods l1, l2, l3.

l1 is a likelihood of sampling "fun" given a sequence "Data Science is", with current model params.

l2 is a likelihood of sampling "cool" given a sequence "Machine Learning is", with current model params.

l3...

Then, we get a product of them, which is the likelihood of all the "events" together.

l_batch = l1 * l2 * l3

Finally, we can apply the negative logarithm and get the "negative log-likelihood" loss function.

-log(l_batch)

Theoretically, we could drop the negative logarithm and optimize our joint likelihood (l_batch = l1 * l2 * l3) directly. It would be maximization instead of minimization, because of missing minus sign. It's perfectly differentiable and backpropagation would work. The only problem would be our likelihoods product could be extremely small and cause arithmetic problems or at least be inconvenient to read and work with.

But we could use the sum of likelihoods instead of the product, and maximize it. It would be differentiable too.

l1 + l2 + l3

The question is a) if I reason correctly b) why engineers/scientists don't use the sum of likelihoods instead of product as a loss function for neural network classification then? (we could include a minus sign, to turn it into a minimization problem and be consistent with other loss functions)

$\endgroup$
15
  • 2
    $\begingroup$ Can you edit your question to give a toy illustration of what you mean? I do not see how using sum would "optimize the likelihood of separate data samples". $\endgroup$
    – Henry
    Commented Feb 2 at 11:03
  • $\begingroup$ “But our goal here is not to follow the probability theory concepts.” this sounds weird, why not? “Our goal is to optimize the likelihood of separate data samples.” ok, maybe you are after some more complicated scheme (it is confusing because optimizing likelihood is still a probability theory concept; what is different?). How do you wish to optimize this seperately, if not by using multiplication of likelihoods? Are you considering some different cost function or different assumptions about indepdence? Can you explain this more exactly? $\endgroup$ Commented Feb 2 at 12:26
  • 1
    $\begingroup$ From a practical perspective, the sum of likelihoods is probably going to be much less convex than the product. For example, if your data have Gaussian likelihoods, with a sufficiently small variance, you will have one peak for each datapoint, even in linear regression. $\endgroup$ Commented Feb 2 at 21:51
  • 1
    $\begingroup$ The justification for an MLE is that you're maximizing the joint probability of the model parameters given the data. I'm not aware of any justification for simply summing the probability of each observation. You can add together the probabilities of mutually exclusive events, but that's not the scenario here -- events need not be mutually exclusive, and you'll easily end up with probabilities larger than 1, which departs from the standard probability axioms. $\endgroup$
    – Sycorax
    Commented Feb 2 at 22:01
  • 1
    $\begingroup$ Which is more likely, two independent events occurring with probabilities of 99% and 1% or two independent events occurring with probabilities of 20% and 20%? The former occurs less than 1% of the time, the latter 4% of the time, but the former probabilities add to 100% while the latter only add to 40%. Probabilities multiply, not add. $\endgroup$
    – jbowman
    Commented Feb 2 at 23:36

4 Answers 4

4
$\begingroup$

Commonly, in statistics, we make a strong distributional assumptions about where each datapoint comes from: we say that $P(y_i|\theta) = f(y_i,\theta)$. If we make the assumption that the data are independent, then we can get the joint probability of all our data by taking the product of each density: $P(\{y_1, \ldots, y_N\}|\theta) = \prod_{i=1}^N f(y_i|\theta)$.

When it comes time to find out what the parameters $\theta$ are, a common approach is maximum likelihood. This involves simply computing the maximum of $P(\{y_1, \ldots, y_N\}|\theta)$ now viewed as a function of $\theta$:

$$ \hat\theta = \underset{\theta}{\textrm{argmax}} \prod_{i=1}^N f(y_i,\theta) $$

So this is the first part of an answer to your question: if we are set on doing maximum likelihood, the product comes straight from our assumption of independence and basic probability rules.

But you ask: is it possible to consider the sum of the probabilities instead? This would involve abandoning the idea of maximum likelihood (but it could probably fall under the umbrella of M-Estimation):

$$ \hat\theta = \underset{\theta}{\textrm{argmax}} \sum_{i=1}^N f(y_i,\theta) $$

One downside is that it's not as well motivated as the product formulation, because that's based on the probability of the entire sample rather than being some function of the individual probabilities. But nevertheless, this cost functional is still monotonically increasing in each individual probability, right? So it's not a priori a ridiculous thing to try, and I imagine that there may be circumstances in which it does reasonable things.

However, in the most common circumstances, it's going to have inferior properties to the product formulation. Let's do a simple example. Imagine that we have some data, and we want to fit a normal distribution to it with a fixed variance, say $\sigma=0.1$, but want to find the mean $\mu$ given this fixed variance assumption:

$$ y_i \overset{iid}{\sim} N(\mu, 0.1^2) $$

We can compute the individual probabilities, and then get the standard (product) likelihood versus the proposed sum functional.

Here's code for that in R:

N <- 10
x <- rnorm(N)

sigma <- 0.1

indiv_lik <- function(mu) dnorm(x, mu, sigma)
mu_seq <- seq(-3,3,length.out=10000)

par(mfrow=c(2,1))
plot(mu_seq, sapply(mu_seq, function(mu) sum(indiv_lik(mu))), type = 'l')
plot(mu_seq, sapply(mu_seq, function(mu) prod(indiv_lik(mu))), type = 'l')

This is the plot produced; the top plot shows the sum formulation, the bottom the standard product formulation:

sum is much more nonconvex

Even though summation and product-taking are conceptually similar, the plots look nothing alike! The sum plot is highly multimodal and nonconvex, while the bottom plot is unimodal. And this is just for a normal likelihood with known variance!

The reason is that the normal likelihood is log-convex, which is a property conserved under product-taking.

However, the property conserved under summation is not log-convexity, but just plain-ol' convexity, which the normal density does not possess. The same is true of many other popular distributions: they are much friendlier when combined via a product than a sum.

Now all this being said, could I rule out some circumstance where summing the probabilities leads to a better inference situation than producting it? I certainly don't know of a proof ruling this out. But in most common situations, it will be inferior.

$\endgroup$
1
$\begingroup$

No, you cannot simply drop the negative logarithm. This will give you a completely different loss function with unpleasant properties, not only in practice, but also in theory.

Assume we make a probabilistic prediction, i.e. we come up with a density $f$ (or $f_\theta$ for some parameter $\theta$, if you prefer a parametric family) for some random variable $Y$ with unknown density. We can then evaluate our prediction based on an observation $y$ via the log-loss $$ L_\mathrm{log}(f, y) = - \log f(y) $$ which is closely tied to the maximum-likelihood principle and has the advantage that it is a strictly proper scoring rule. This means that if $Y$ has distribution $f$, then $$ \mathbb{E}_{Y \sim f} L_\mathrm{log} (f, Y) \le \mathbb{E}_{Y \sim f} L_\mathrm{log}(g, Y) $$ for all probability densities $g$. Equality holds if and only if $f=g$. In other words, $L_\mathrm{log}$ gives on average lowest loss to the correct prediction.

Now if we followed your proposition, we could define the alternative loss function (called linear score in the reference below) $$ L_\mathrm{lin}(f, y) = - f(y) $$ and use this for evaluation. However, this choice is not a proper scoring rule and will not be minimized by the true data distribution in expectation. Gneiting and Raftery (see reference below) illustrate this for the Gaussian case and state that

the linear score encourages over-prediction at the modes of an assessor’s true predictive density.

Hence, $L_\mathrm{lin}$ will prefer predictions which concentrate too much around the mode of the true data generating distribution. It is thus a poor choice of loss function for training or evaluation of your models and log-loss should always be preferred.

Reference

Gneiting & Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation

$\endgroup$
0
$\begingroup$

This relates to the questions: Could a mismatch between loss functions used for fitting vs. tuning parameter selection be justified? and If the predicted value of machine learning method is E(y | x), why bother with different cost functions for y | x?

The likelihood function is popular because it relates to filtering statistical noise in an efficient way. The likelihood does not need to represent some function related to the actual final goal that is to be optimized. Many different functions may be used for that. But even when it is not the same as that final cost function, by using the likelihood one obtains an estimate/fit that has good performance on that final cost function (example: when estimating the mean of a Gaussian population, minimizing the sum of squared residuals will lead to an estimate that minimizes the absolute error).

In the case of a complex model like a language learning model, the parameters of the model may be very much unrelated and the fitting of all of them together in a single value of a single likelihood function may indeed not be necessary or it may be even undesirable. And I also don't think that many people take that approach, ie. for complex models more complex cost functions are incorporated into the training of such models. For example, the fitting of different aspects of the model are weighted according to desired relative importance of these aspects, in order to optimize the variance/error and performance in some final cost function. (Still, using the product in some may remains desirable because it makes computations easier. The relationship between the least squares method and the Gaussian error distribution is a good example)

Also related: When combining p-values, why not just averaging?

$\endgroup$
0
$\begingroup$

The "likelihood" is a derivative concept. The primary concept is probabilistic: we consider the joint density of a sample of independent observations, because it expresses the probability of this sample actually being realized as a whole.

So by determining the parameters that maximize the joint density/likelihood, we find the estimates that maximize the probability of this sample happening, given the data and the assumption we have made about the statistical distribution family (with unknown parameters) that characterizes the data. This is why, comparing models (i.e. different assumptions about the distribution), in terms of the attained sample likelihood value is meaningful: the model that gives the higher likelihood value, is more probable than the rest.

This is how, and why, "maximum likelihood estimation" was originally conceived -not as some mathematical maximization algorithm, but based on probabilistic concepts.

An underlying concept exists also for the "least-squares" estimation method, where we minimize the sum of squared deviations: it makes sense to obtain the estimates that minimize the sum of deviations from the actual sample values. Here we minimize the inaccuracy.

So let's say we optimize the sum of likelihoods... We have no argument to support and argue in favor of the estimates thus obtained, as being good estimates of the unknown true values of the parameters. They do optimize some function (the sum of individual ikelihoods in this case), but then we could form some other function of the data (among myriads), and obtain a different set of estimates. How we would be able to compare the two and conclude which on is the preferable one?

Of course, as mathematical transformations go, any strictly monotonic transformation of the joint density / product of individual likelihoods would give us the same estimates as the optimization of the joint likelihood, but here too, we base our estimation to the underlying probabilistic concept, and then we apply mathematics to make our life easier (like taking the logarithm for example).

To summarize: we do not use the sum of likelihoods, because it is disassociated from probabilistic concepts, and hence it is just an arbitrary function to optimize, providing no support or comfort that the estimates obtained will be adequate.

Certainly, one could imagine a project where we just try out various arbitrary functions to optimize, and through simulations, checking whether they perform better than the joint likelihood... but this would be a never-ending project, since each such arbitrary function should be tried against very many distributions and very many parameter values, so that the whole project is comprehensive enough to be persuasive... too much cost, and what would the benefit be?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.