When learning a model $\theta$, what is bad about maximize $\sum_{i}{p(x_{i}|\theta)}$ instead of maximizing $\prod_{i}{p(x_{i}|\theta)}$?

In a typical learning setting, a model is trained to maximize likelihood. Likelihood is probability of data given a model, and with IID assumption, it can be written as $\prod_{i}{p(x_{i}|\theta)}$. This is what we usually do in statistics and machine learning.

Here is an instructive question: Why shouldn't it be a summation of probabilities?

I think chaining questions can be raised and it's worth contemplating upon.

  • How bad if we perform learning with such a criterion?
  • Is there any pathological or illustrative examples?
  • Is ML always superior than the not-well-formulated sum-of-probabilities?
  • 4
    As you say yourself, "likelihood is probability of data given a model, ...". One of the answers you seek comes from explaining the second part of this statement better. The probability we are interested in is the joint probability of observing all the data together. Given the IID assumption, this is the product of the marginal probabilities: $\prod_{i}{p(x_{i}|\theta)}$. Thus, by definition you can see why we do not maximise the sum of the marginal probabilities. – Alex Aug 31 '16 at 6:46
  • @Alex Thanks for the comment. In your words, I think my question is "why should we maximize the joint probability?" This question is reasonable because in machine learning there are many other frameworks/objectives that are not relevant with probabilistic framework. For example, the Bayesian approach does not maximize the joint prob. Plus, I am wondering if there is a particular example that sum-of-probs would fail. – Sangwoong Yoon Aug 31 '16 at 8:26
  • Intuitively, we maximize the likelihood because that corresponds to the parameters with the best chance of producing the sample we got (why would you consider parameters unlikely to produce your sample?). While this isn't the only possible criterion, it's one with a host of important properties (some addressed in a number of questions on site). There's no corresponding interpretation nor any similar collection of useful properties.to go with it for adding the probabilities. ... ctd – Glen_b Aug 31 '16 at 10:50
  • ctd... Presuming you don't wish to advocate randomly generating criteria in the hope that one might eventually be useful in some situation, what would be a reason to consider it? – Glen_b Aug 31 '16 at 10:54
  • 1
    I confess to having a negative reaction to questions of the form "why won't this procedure I made up work" unless they include a good justification for considering such a procedure in the first place. So I would ask you: can you supply a reason why maximizing the sum of probabilities would have any kind of meaning at all? A good reason would be grounded in statistical theory or the axioms of probability. – whuber Aug 31 '16 at 14:07
up vote 7 down vote accepted

$P(x_i|\theta)$ is the probability that you observe $x_i$ for a given parameter value $\theta$. In a sample you observe several $x_i, i=1,2,\dots,n$ simultanuously and (assuming independence) the probability that you observe all the values in your sample simultanuously is $\prod_i P(x_i|\theta)$. So maximising this with respect to $\theta$ means that you look for the value of $\theta$ that makes your (full) observed sample most probable.

$\sum_i P(x_i|\theta)$ is (assuming that that the observations do not intersect) would be the probability that you observe at least one of the $x_i$. So if you maximise this then you look for the value of $\theta$ such that at least one of the $x_i$-values is observed.

As you have observed all the $x_i$ values, it makes sense to compute the probability that you all observed these simultanuously I think. This is also what @Alex says in his comment on the ''joint probability''.

As @Tim argues (+1) there are reasons to log-transform it and as the 'log' is monotic you will find the same $\theta$.

Due to your comment ''the Bayesian approach does not maximize the joint prob'' I make the following remark:

if you observe one value $x_1$ then the Bayes approach would compute the posterior $P(\theta|x_1)\propto P(x_1|\theta) p(\theta) $.

If I observe a second observation $x_2$ then I will update my estimate using the ''updated prior'', so you compute $P(\theta|x_2)\propto P(x_2|\theta) p^{upd}(\theta) $ and that prior would certainly be the one that already takes into account the other (independent) observations ($x_1$), so $p^{upd}(\theta)$ would be $P(\theta|x_1)$ and the new posterior becomes $P(\theta|x_2)\propto P(x_2|\theta) P(x_1|\theta) p(\theta) $.

After observing the full sample you find the posterior $P(\theta|x)\propto \prod_i P(x_i|\theta) p(\theta) $, so what you say: ''The Bayesian apporach does not maximize the joint prob'' is not fully correct; the Bayesian approach is also using the joint probability of the full sample, i.e. $\prod_i P(x_i|\theta)$, in the likelihood function.

  • Thank you very much for the clear answer. I liked the explanation that "the probability that you observe at least one of the x_i." I do aware that Bayesian is of course related to the joint probability. I mentioned it in order to emphasize the joint prob is not the only possible criterion. – Sangwoong Yoon Aug 31 '16 at 9:53
  • You're welcome:-) If you like it very much you may also ''accept'' it as your prefered answer – user83346 Aug 31 '16 at 9:55

As you already noticed, maximum likelihood is used when we are dealing with independent and identically distributed random variables. By the definition of independent random variables

$$ p_{X,Y}(x, y) = p_X(x)\,p_Y(y) $$

So if we are interested in the joint distribution of independent random variables, then by definition, we need to multiply them. However, you can always take the sum of log-probabilities to obtain log-likelihood given the properties of logarithms, specifically

$$ \log\ a + \log\ b = \log\ (ab) $$

In fact, using log-likelihoods is preferable to using "raw" likelihoods because they behave better when optimizing and due to underflow issues.

One of the biggest problems I can think of is that you become very likely to generate samples that do not exist in the data you generated your model from.

An example could be if you made a model that generates mammals based on their features such as whether they have a tail, how many eyes they have and the number of legs.

Most mammals have 4 legs, so let's assign the probability 0.9 to that feature and a small subset (humans and monkeys I guess) have 2 legs, so they get a probability of 0.1 and the probability for any other number of legs is 0.0. As far as I know there are no mammals with more or less than 2 eyes, so let's just assign the probability 1.0 to that and 0.0 to everything else.

      1 | 2 | 3 | 4 | 5 |...
legs 0.0|0.1|0.0|0.9|0.0|0.0 
eyes 0.0|1.0|0.0|0.0|0.0|0.0

Now, if you are maximizing over the product of probabilities you will find that the probability of generating an animal with 2 eyes and 4 legs is 0.9, 2 eyes and 2 legs is 0.1 whereas generating an animal with 2 eyes and 10 legs is 0.0, so you're staying pretty close to observations you would see in your training set.

$$ p(legs=4, eyes=2) = 0.9\\ p(legs=2, eyes=2) = 0.1\\ p(legs=5, eyes=2) = 0.0 $$

If you instead maximize over the sum of probabilities a mammal with 2 eyes and 4 legs will get a value of 1.9, 2 eyes and 2 legs 1.1 and a mammal with 2 eyes and 3 or even 400 legs gets a value of 1.0.

$$ f(x) = \sum p(x)\\ f(legs=4, eyes=2) = 1.9\\ f(legs=2, eyes=2) = 1.1\\ f(legs=5, eyes=2) = 1.0 $$

The problems here are of course that first of all these values can no longer be interpreted as probabilities as they are no longer constrained to [0, 1], but also that samples not observed in the training set are almost as likely as observed but rare samples.

Maybe there are cases where generating one probable feature makes outlandish values for other features likely, but nothing immediately springs to mind.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.