The difference between the Bayes Classifier and The Naive Bayes Classifier?

Question

I'm trying to find the connection between both classifiers. In NBC we assume that all the features are independent of each other so we can calculate the posterior probability easier. I assume Bayes Classifier is more complex but how is the process different from NBC?

Answer 1

Naive Bayes assumes conditional independence, $P(X|Y,Z)=P(X|Z)$, Whereas more general Bayes Nets (sometimes called Bayesian Belief Networks) will allow the user to specify which attributes are, in fact, conditionally independent. There is a very good discussion of this in Tan, Kumar, Steinbach's Introduction to Data Mining textbook. They also have instructor powerpoint slides: Tan, Steinbach, Kumar - Introduction to Data Mining - Lecture Notes for Chapter 5 - Data Mining Classification: Alternative Techniques which should give an example of this assumption and why it can be flawed.

Answer 2

How would a "non-naive" Bayesian classification work?

Each record is classified as:

1. Find all records that have exactly the same features
2. Determine what class of those records is most common
3. Assign the most common class to the record

Problem: if there are many features, it is unlikely to find records that match exactly the same features.

Solution: Naive Bayes classifiers compute therefore the probabilities of each feature occurring in a class independently.

Answer 3

For the Bayesian network as a classifier, the features are selected based on some scoring functions like Bayesian scoring function and minimal description length(the two are equivalent in theory to each other given that there are enough training data). The scoring functions mainly restrict the structure (connections and directions) and the parameters(likelihood) using the data. After the structure has been learned the class is only determined by the nodes in the Markov blanket(its parents, its children, and the parents of its children), and all variables given the Markov blanket are discarded.

For the Naive Bayesian Network which is more well-known nowadays, all features are considered as attributes(all attributes are part of the class variable Markov blanket) and are independent given the class.

Bayesian networks and naive Bayesian network have their own advantages and disadvantages and we can see the performance comparison(done on 25 data sets mainly from the UCI repository) as depicted below:

We can see that there are some points below the diagonal line representing the Naive Bayes performs better than the Bayesian Network on those datasets and some points above the diagonal line representing the reverse on some other datasets.

Bayesian Network is more complicated than the Naive Bayes but they almost perform equally well, and the reason is that all the datasets on which the Bayesian network performs worse than the Naive Bayes have more than 15 attributes. That's during the structure learning some crucial attributes are discarded.

We can combine the two and add some connections between the features of the Naive Bayes and it becomes the tree augmented Naive Bayes or k-dependence Bayesian classifier.

References:
1. Bayesian Network Classifiers