4
$\begingroup$

I had the idea that we can overcome the conditional independence of features within Naive Bayes classification by assuming that we have latent (hidden) sub-classes. Let me explain.

For example, if we have this features vector

(1, 1, 0, 1, 0)

in Naive Bayes we would calculate probability of this vector given a class like this:

$ p_1 \cdot p_2 \cdot (1 - p_3) \cdot p_4 \cdot (1 - p_5), $

where $p_i$ is probability of feature to be present (value 1) given the class.

However, as a generalization, we can say that within a given class we have two sub-classes. In this case probability of the above given vector will be calculated as:

$ p_1 \cdot p_2 \cdot (1 - p_3) \cdot p_4 \cdot (1 - p_5) \cdot \nu_1 + p'_1 \cdot p'_2 \cdot (1 - p'_3) \cdot p'_4 \cdot (1 - p'_5) \cdot \nu_2, $

where $p_{i}$ ans $p'_{i}$ are probabilities of the feature $i$ to be present given the first and the second sub-classes, respectively. $\nu_j$ are the probabilities o the two sub-classes.

I assume that this method was already introduced and considered in details. So, my question is what is the name of the method and what is efficient way to calculate its parameters based on a given data set?

$\endgroup$
3
  • $\begingroup$ Why not just use logistic regression $\endgroup$
    – seanv507
    Jun 14, 2023 at 14:23
  • $\begingroup$ @seanv507 because logistic regression assumes very simple (linear) dependency. $\endgroup$
    – Roman
    Jun 15, 2023 at 9:53
  • 1
    $\begingroup$ Linear in parameters doesn't mean you can't add arbitrary feature transformations: interactions, logs etc $\endgroup$
    – seanv507
    Jun 15, 2023 at 16:30

1 Answer 1

2
+25
$\begingroup$

The idea that each class is composed of multiple sub-classes is called "Latent Class Naive Bayes". This is an extension of the standard Bayes approach that adds the possibility for latent/hidden sub-classes to model dependencies between features.

Some interesting papers I found regarding this topic:

An algorithmic approach to analyze the data this way is the Expectation-Maximization (EM) algorithm, which is an iterative optimization algorithm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.