What is bad about maximizing $\sum_{i}{p(x_{i}|\theta)}$ instead of maximizing $\prod_{i}{p(x_{i}|\theta)}$

Question

When learning a model $\theta$, what is bad about maximize $\sum_{i}{p(x_{i}|\theta)}$ instead of maximizing $\prod_{i}{p(x_{i}|\theta)}$?

In a typical learning setting, a model is trained to maximize likelihood. Likelihood is probability of data given a model, and with IID assumption, it can be written as $\prod_{i}{p(x_{i}|\theta)}$. This is what we usually do in statistics and machine learning.

Here is an instructive question: Why shouldn't it be a summation of probabilities?

I think chaining questions can be raised and it's worth contemplating upon.

• How bad if we perform learning with such a criterion?
• Is there any pathological or illustrative examples?
• Is ML always superior than the not-well-formulated sum-of-probabilities?

Answer 1

$P(x_i|\theta)$ is the probability that you observe $x_i$ for a given parameter value $\theta$. In a sample you observe several $x_i, i=1,2,\dots,n$ simultanuously and (assuming independence) the probability that you observe all the values in your sample simultanuously is $\prod_i P(x_i|\theta)$. So maximising this with respect to $\theta$ means that you look for the value of $\theta$ that makes your (full) observed sample most probable.

$\sum_i P(x_i|\theta)$ is (assuming that that the observations do not intersect) would be the probability that you observe at least one of the $x_i$. So if you maximise this then you look for the value of $\theta$ such that at least one of the $x_i$-values is observed.

As you have observed all the $x_i$ values, it makes sense to compute the probability that you all observed these simultanuously I think. This is also what @Alex says in his comment on the ''joint probability''.

As @Tim argues (+1) there are reasons to log-transform it and as the 'log' is monotic you will find the same $\theta$.

Due to your comment ''the Bayesian approach does not maximize the joint prob'' I make the following remark:

if you observe one value $x_1$ then the Bayes approach would compute the posterior $P(\theta|x_1)\propto P(x_1|\theta) p(\theta) $.

If I observe a second observation $x_2$ then I will update my estimate using the ''updated prior'', so you compute $P(\theta|x_2)\propto P(x_2|\theta) p^{upd}(\theta) $ and that prior would certainly be the one that already takes into account the other (independent) observations ($x_1$), so $p^{upd}(\theta)$ would be $P(\theta|x_1)$ and the new posterior becomes $P(\theta|x_2)\propto P(x_2|\theta) P(x_1|\theta) p(\theta) $.

After observing the full sample you find the posterior $P(\theta|x)\propto \prod_i P(x_i|\theta) p(\theta) $, so what you say: ''The Bayesian apporach does not maximize the joint prob'' is not fully correct; the Bayesian approach is also using the joint probability of the full sample, i.e. $\prod_i P(x_i|\theta)$, in the likelihood function.

Answer 2

As you already noticed, maximum likelihood is used when we are dealing with independent and identically distributed random variables. By the definition of independent random variables

$$ p_{X,Y}(x, y) = p_X(x)\,p_Y(y) $$

So if we are interested in the joint distribution of independent random variables, then by definition, we need to multiply them. However, you can always take the sum of log-probabilities to obtain log-likelihood given the properties of logarithms, specifically

$$ \log\ a + \log\ b = \log\ (ab) $$

In fact, using log-likelihoods is preferable to using "raw" likelihoods because they behave better when optimizing and due to underflow issues.

Answer 3

One of the biggest problems I can think of is that you become very likely to generate samples that do not exist in the data you generated your model from.

An example could be if you made a model that generates mammals based on their features such as whether they have a tail, how many eyes they have and the number of legs.

Most mammals have 4 legs, so let's assign the probability 0.9 to that feature and a small subset (humans and monkeys I guess) have 2 legs, so they get a probability of 0.1 and the probability for any other number of legs is 0.0. As far as I know there are no mammals with more or less than 2 eyes, so let's just assign the probability 1.0 to that and 0.0 to everything else.

      1 | 2 | 3 | 4 | 5 |...
legs 0.0|0.1|0.0|0.9|0.0|0.0 
eyes 0.0|1.0|0.0|0.0|0.0|0.0

Now, if you are maximizing over the product of probabilities you will find that the probability of generating an animal with 2 eyes and 4 legs is 0.9, 2 eyes and 2 legs is 0.1 whereas generating an animal with 2 eyes and 10 legs is 0.0, so you're staying pretty close to observations you would see in your training set.

$$ p(legs=4, eyes=2) = 0.9\\ p(legs=2, eyes=2) = 0.1\\ p(legs=5, eyes=2) = 0.0 $$

If you instead maximize over the sum of probabilities a mammal with 2 eyes and 4 legs will get a value of 1.9, 2 eyes and 2 legs 1.1 and a mammal with 2 eyes and 3 or even 400 legs gets a value of 1.0.

$$ f(x) = \sum p(x)\\ f(legs=4, eyes=2) = 1.9\\ f(legs=2, eyes=2) = 1.1\\ f(legs=5, eyes=2) = 1.0 $$

The problems here are of course that first of all these values can no longer be interpreted as probabilities as they are no longer constrained to [0, 1], but also that samples not observed in the training set are almost as likely as observed but rare samples.

Maybe there are cases where generating one probable feature makes outlandish values for other features likely, but nothing immediately springs to mind.