Let us consider a problem of binary classification based on use of several nominal (categorical variables). For example, we would like to predict if a person has a car based on his/her gender, education and marital status.

Having the data we can estimate probability of belonging to a category (having a car) given each of the variables separately. For example, we can estimate how probable is that a person has a car given his/her gender. Or, how probable that the person has a car given his/her education.

Then for a given person we have values of all attributes (gender, education, marital status) and we can calculate probabilities of having a car corresponding to the given values.

For example, we know that the person is male and given that, the probability of having a car has to be 0.6. We also know that a person has high education and for people from this category probability of having a car is 0.4. Finally, we know that a person is married and among all married persons the probability to have a car is 0.9. So, now we have three probabilities (0.6, 0.4 and 0.9) and we need to combine them somehow to get the final (more accurate) estimation of the probability.

What is the right way to do that? The simplest (and the most "natural") option would be to calculate an average or a weighted average of these probabilities:

$P(C | f_1 \cdot f_2 \dots f_n) = \omega_1 \cdot P(C | f_1) + \omega_2 \cdot P(C | f_2) + \omega_n \cdot P(C | f_n)$

However, Bayesian approach gives another answer. The probability to be in a category given the "features" is given by:

$P(C | f_1 \cdot f_2 \dots f_n) = \frac{P(f_1 \cdot f_2 \dots f_n | C) \cdot P(C)}{P(f_1 \cdot f_2 \dots f_n)}$

Now let us assume that all the features are independent for any category and get the following:

$P(C | f_1 \cdot f_2 \dots f_n) = \frac{P(f_1 | C) \cdot P(f_2 | C) \dots P(f_n | C) \cdot P(C)}{P(f_1) \cdot P(f_2) \dots P(f_n)}$

The above expression can be rewritten as follows:

$P(C | f_1 \cdot f_2 \dots f_n) = \frac{P(f_1 | C) \cdot P(C)}{P(f_1)} \cdot \frac{P(f_2 | C) \cdot P(C)}{P(f_2)} \dots \frac{P(f_n | C) \cdot P(C)}{P(f_n)} \cdot P^{1-n} (C)$

Which is the same as

$P(C | f_1 \cdot f_2 \dots f_n) = P(C | f_1) \cdot P(C | f_2) \dots P(C | f_n) \cdot P^{1-n} (C)$

So, in the end we have a product of different probabilities (predictions) multiplied by $P^{1-n} (C)$.

So, basically my question is: What is the intuition behind this formula (if it is right). Why do we multiply probabilities. For example, we can take a logarithm of probabilities and then we will get a sum instead of a product. It is already closer to the average but still it is not an average of logarithmic probabilities. It will be just their sum minus some value.


I see that I made a logical mistake. I have assumed that features are independent for any given category. Then I thought that if features are independent for any category, then they have to be independent in general. However, this conclusion is wrong. Features are independent withing any given category but they are dependent through category (so, they are not independent in general but only withing each category).

So, it looks like Bayesian approach (under assumption of independence within any category) tells us to combine the probabilities in the following way:

$P(C | f_1 \cdot f_2 \dots f_n) = P(C | f_1) \cdot P(C | f_2) \dots P(C | f_n) \cdot \frac{P(C) \cdot P(f_1) \cdot P(f_2) \dots P(f_n)}{P(f_1 \cdot f_2 \dots f_n) P^{n} (C)}$

But my question remains. Is there intuition behind the above expression? For example, we have three features (attributes) as before and they "generate"the following predictions (probabilities): 0.4, 0.5, 0.6. In this case I would expect that the "combined" probability has to be around 0.5 but by multiplying all three probabilities we get: 0.12 which is too small. And this problem of small value of the product is even larger if number of features is larger. But of course we correct these small values of the product of probabilities by using the following factor:

$\frac{P(C) \cdot P(f_1) \cdot P(f_2) \dots P(f_n)}{P(f_1 \cdot f_2 \dots f_n) P^{n} (C)}$

So, my question is: How one can easily see why this factor provides a good correction?

  • $\begingroup$ IMO, Eq 4 is false with regards to Eq. 3. I do not see how you factorize $P(C)$ $n-1$ times. Maybe you wanted to set $1/P^{n-1}(C)$ ? $\endgroup$ – peuhp Dec 15 '15 at 11:09
  • $\begingroup$ Yes, you are right. Thank you for the remark. I have corrected the question. $\endgroup$ – Roman Dec 15 '15 at 11:13

I think that something is inconsistent in your model. In particular you cannot factorize the denominator of your bayes formula: $$ p(f_1,\ldots,f_n) \ne p(f_1) \ldots p(f_n) $$ Indeed by definition of your model: $$ p(f_1,f_2)= \int p(f_1,f_2|C) p(C) dC = \int p(f_1|C) p(f_2|C) p(C) dC $$ and for $i$ in 1,2: $$ p(f_i)=\int p(f_i|C) p(C) dC $$ and I see no reason that:

$$ \int p(f_1|C) p(f_2|C) p(C) dC = \int p(f_1|C) p(C) dC \cdot \int p(f_2|C) p(C) dC $$

More intuitively, if all $f_i$ depends from the value of $C$ it should look surprising that the $f_i$ are independent regardless from the value of $C$.

Nevertheless, in the case where $p(f_i|C)=p(f_i)$ and thus $p(f_1,f_2)=p(f_1)p(f_2)$, your last equation simplifies to $P(C|f_1,f_2)=P(C)$ which seems logical.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.