2
$\begingroup$

I am experimenting with the Bayesian Naive Classifier and I have come across a situation for which I wasn't able to find a solution on the net.

Suppose you have a dataset with a few attributes that takes real values. The values of the observations varies around a mean as they typically do -- except for a particular attribute and for a particular class where all observations have the same value. For that particular attribute-class subset, the variance will be 0 and the probability density for such a distribution will be infinity for any 'x'.

How should a BNC handle such situations? Should we discard an attribute if it has a degenerate (zero variance) distribution for any class even if it has a normal distribution otherwise?

Edit:

I'd rather not ignore them if possible. They may have a lot of information, just not for the class for which variance is 0. I'll try to explain myself:

Suppose you have a bowl of cookies, some of them were made by a machine and the others were handmade and you want to classify them (as well as eating them). One of the attributes you use to classify is the diameter of the cookie. A handmade cookie will have a good variance while the machine-made ones all have the same (0 variance).

Say all machine-made cookie are all 2.5 inches in diameter. I agree that picking a 2.5 inch cookie at random will give us no useful information; but picking a 3 inch cookie is a pretty strong indication that is was handmade.

In the case of a 0 variance distribution, I was thinking making the Probability Density function return a 1.0 (neutral) when 'x' is the mean and 0.0 when it's not. But I'm not experienced enough in statistics to understand the possible side-effects of this.

$\endgroup$

2 Answers 2

0
$\begingroup$

The PDF is the probability density function. What you're describing doesn't have a density of 1 at the mean, it has a probability mass of 1 at the mean. If you really are certain that the variance of machine cookies is zero, then you should classify all cookies that don't have the mean of machine cookies as homemade cookies with probability 1. If a cookie does have the mean of machine cookies, then your probability of being a homemade cookie should be prior * x/(prior * x+1-prior), where x is the probability mass of (diameter=machine mean|homemade). If you have infinite precision, x=0. Otherwise, if you have an interval I such that if the actual diameter is in I, the measured diameter will be the machine mean, then x is the integral of the PDF of the diameter for homemade cookies over I.

As for where the formula prior * x/(prior * x+1-prior) comes from:

The number of homemade cookies with a machine mean is proportional to the prior probability times the conditional probability, prior * x. For machine cookies, the conditional probability is 1, so it's just the prior probability of machine, which is 1-(prior probability of homemade).

Often, when learning about conditional probabilities, it's easier to work with proportions rather than percentages (that is, "10 out of 100" is more concrete than "10%"). If you're having trouble following an argument, putting in integers and taking successively smaller portion of the total as more conditions are added can help.

For example, suppose there are 1000 cookies, and 20% of the total number of cookies are homemade. And suppose that 10% of homemade cookies have the same diameter as machine cookies. Then the number of homemade cookies with the machine mean is 1000*10%*20% = 20. Since 20% are homemade, 80%, or 800, are machine made. So there are 20+800 cookies with machine mean, and of them, 20 are homemade. So if you know a cookie has machine mean, the probability of it being homemade is 20/(20+800) = 2.4%.

Now, again, I introduced the 1000 just so that we would have integer cookies and the math would be easier to follow; the 1000 should of course cancel out, so if you take the formula

prior * x /(prior * x+1-prior)

and substitute in prior = 20%, x = 10%, you get

.2*.1/(.2*.1+.8) = .02/.82 = .024

$\endgroup$
1
  • $\begingroup$ Thanks a lot @Accumulation. I would be interested in knowing why the probability of being a homemade cookie should be prior * x/(prior * x+1-prior). Please can you tell me where this comes from? (Sorry if it's a basic stats question, but I'm pretty much a beginner) $\endgroup$ Apr 13, 2018 at 21:17
1
$\begingroup$

Within statistics, all is uncertain, also the class membership of each pattern. Probabilities that are purely 0 and 1, that indicates a deterministic mechanism, for which statistics was not developed. As soon as uncertainty comes into play, a deterministic paradigm is left and it becomes statistics and probability theory.

When you can with 100% certainty (on a very large, representative dataset) classify all patterns correctly, you get into the deterministic situation. Boolean algebra with truth-predicates and (in)equalities are sufficient to represent the classification rule.

$\endgroup$
2
  • $\begingroup$ Thanks Match Maker. If I understand correctly, you say that in situations like the cookie problem, I don't need a classifier; a if() would do just fine. $\endgroup$ Apr 13, 2018 at 18:50
  • $\begingroup$ Correct - that is the point I wanted to make. $\endgroup$ Apr 13, 2018 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.