1
$\begingroup$

The following is a formula that finds the posterior probability of a class (i.e. yes or no) given four conditionally independent attributes:

$$P(c|X) = P(x_1|c)\cdot P(x_2|c)\cdot P(x_3|c)\cdot P(x_4|c)\cdot P(c).$$

Found here: Naive Bayesian

In my experience, I've only seen formulas showing the posterior probability of attributes given a class, such as the Bayes Theorem formula:

$$P(B_j|A) = \frac{P(A|B_j)\cdot P(B_j)}{\sum_i P(A|B_i)P(B_i)}.$$

Grabbed from here: 1.4.3 Bayes' Rule

Is the first formula a derivation of the second? If so, how is it derived?

If not, it must have some relation, but I can't discern what it is.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Initially, there is little context for why the author inserted that formula there, so it is challenging to figure out what its purpose is. The formula's equivalence is made possible by the 'chain rule' given 'conditional independence' of the attributes. Open link and see slide 20:

Probability, Conditional Probability & Bayes Rule

The expression on the right hand side and its compliment expression is used to determine the probability of each class (binomial in this case) c and not-c given X (i.e. given all four attributes) further on down the website's page, though I didn't find it obvious. Then the class with the higher attribute becomes the classification for the iteration of those four attributes.

The author inserts:

$P(c|X) = P(x_1|c)\cdot P(x_2|c)\cdot P(x_3|c)\cdot P(x_4|c)\cdot P(c)$

right after he introduces Bayes Theorem in his tutorial, to set us up for Example 2, further down the page, under the subsection called "Example2." The expression and its compliment are used to calculate and compare the probabilities of c ('Yes') and c-not ('No'), given an iteration of four attributes.

The goal is to find the higher probability between

$P(X|c)P(c)$ and $P(X|c^c)P(c^c)$

so we can select the class with higher probability as to classify the four given attributes.

You can see a similar use of this in a subsection call "Bayes Theorem" here:

A practical explanation of a Naive Bayes classifier

Improve this answer
$\endgroup$
4
  • $\begingroup$ The first formula above is not correct, as I mentioned in my answer. For the equality to hold it is necessary to divide the RHS by the "probability of data".The LHS is only proportional to the RHS but is by no means equal. The point is that in actual computations, to compare two posterior probabilities it is sufficient to calculate only the RHS as the "probability of data" is the same for all calculations of posterior classes. In the end, the interest is deciding the most probable class by finding MAP probability, which can be found without division by $p(x)$. $\endgroup$
    – dnqxt
    Commented May 12, 2019 at 4:34
  • $\begingroup$ Ah yes, you are right. Since it was written as an equality. I'll adjust my answer. $\endgroup$
    – spacedustpi
    Commented May 12, 2019 at 4:42
  • $\begingroup$ @dnqxt, I researched this a little more, and the formula is actually correct when the attributes are 'conditionally independent'. Please go to the link I posted in the answer above called 'Probability, Conditional Probability & Bayes Rule' and look at slide 20 to see the proof. I appreciate your contribution to the discussion. Please feel free to add some more if you disagree. If not, please delete your answer. $\endgroup$
    – spacedustpi
    Commented May 14, 2019 at 3:57
  • $\begingroup$ You seem to be confusing conditional independence with Bayes theorem. Note that for the former, P(A,B|C)= P(A|C)P(B|C) but P(C|A,B) is NOT equal to P(C|A)P(C|B). I'll certainly not delete my answer. However, it will get deleted together with the question if you decide to remove the question. $\endgroup$
    – dnqxt
    Commented May 14, 2019 at 5:05
0
$\begingroup$

The first formula is not correct as it misses division by the "probability of data", $P(x)$, which looks like the denominator in the second formula.

The denominator is an expression of the total law of probability, which in case of two complementary events, $B$ and $B^c$, reads: $P(A) = P(A|B)P(B)+ P(A|B^c)P(B^c).$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Initially, I thought they messed up the formula at the "Naive Bayesian" sight too, but it's actually correct. The problem is, there is so little context for why the author inserted that formula there, its difficult to figure out what it's purpose is. It turns out that the formula is modified to determine the probability of each class (binomial in this case) c and not-c given X (given all four attributes). Then the class with the higher attribute becomes the classification for the set of those four attributes. I will post answer below with more explanation and reference. $\endgroup$
    – spacedustpi
    Commented May 12, 2019 at 4:18
  • $\begingroup$ You are right, the formula is incorrect as written, but there is actually a use for the expression on the right hand side, I've adjusted my answer to address your concern. $\endgroup$
    – spacedustpi
    Commented May 12, 2019 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.