What is the extension of Naive Bayes that breaks the conditonal independence?

Question

I had the idea that we can overcome the conditional independence of features within Naive Bayes classification by assuming that we have latent (hidden) sub-classes. Let me explain.

For example, if we have this features vector

(1, 1, 0, 1, 0)

in Naive Bayes we would calculate probability of this vector given a class like this:

$ p_1 \cdot p_2 \cdot (1 - p_3) \cdot p_4 \cdot (1 - p_5), $

where $p_i$ is probability of feature to be present (value 1) given the class.

However, as a generalization, we can say that within a given class we have two sub-classes. In this case probability of the above given vector will be calculated as:

$ p_1 \cdot p_2 \cdot (1 - p_3) \cdot p_4 \cdot (1 - p_5) \cdot \nu_1 + p'_1 \cdot p'_2 \cdot (1 - p'_3) \cdot p'_4 \cdot (1 - p'_5) \cdot \nu_2, $

where $p_{i}$ ans $p'_{i}$ are probabilities of the feature $i$ to be present given the first and the second sub-classes, respectively. $\nu_j$ are the probabilities o the two sub-classes.

I assume that this method was already introduced and considered in details. So, my question is what is the name of the method and what is efficient way to calculate its parameters based on a given data set?

Answer 1

The idea that each class is composed of multiple sub-classes is called "Latent Class Naive Bayes". This is an extension of the standard Bayes approach that adds the possibility for latent/hidden sub-classes to model dependencies between features.

Some interesting papers I found regarding this topic:

• Latent Classification Models (2005)
• Latent class models for classification (2003)

An algorithmic approach to analyze the data this way is the Expectation-Maximization (EM) algorithm, which is an iterative optimization algorithm.