Interpretation of Naive Bayes Probabilities

Question

I'm dealing with a Naive Bayes approach to a Multiclass Classification problem with 9 different classes in the target variable. Let's assume the following: I've fitted a model to my training data and want to apply it to the remaining observations used as test data.

I'm wondering about the interpretation of the calculated probabilities after including information after the estimation of the classifier.

Let's assume the models returns the following hypothetical class probabilities for a single observation in my data: $$ [p_1,p_2,p_3,\ldots,p_9]=[0.3,0.2,0.4,0,\ldots,0,0.1] $$ This would imply a classification to Class 3 without any additional knowledge. Now I get an additional piece of information which tells me that the instance to classify belongs either to Class 1 OR to Class 2.

Other than assigning an observation to the most probable class: Are the probabilities meaningful and interpretable in any way?

Is it possible to assign the instance to Class 1 because I know that it holds the highest probability among the remaining classes if my information is right? The question might be stupid and I have to admit that I haven't actually tried to figure out the math behind this myself but does anyone have an intuition for this?

Answer 1

Quoting my other answer:

Naive Bayes algorithm makes the "naive" assumption, that the features are (conditionally) independent, so by definition of independence

$$ p(x_1, x_2, \dots, x_k \mid y) = \prod_{i=1}^k p(x_i | y) $$

[...] the estimate of conditional probability would be accurate if the assumption of independence holds (and in real life it almost never does).

Saying it differently, the probabilities predicted by naive Bayes algorithm are not well callibrated, so they do not reflect the true probabilities.

On another hand, while the probabilities themselves are not good estimates of the conditional probabilities, they do give you information about relative ordering of the classes. If one class has higher predicted probability, as compared to another, then you should prefer the former as your classification.

Referring to your question, if for classes $c_1, c_2, \dots, c_k$, the algorithm predicted probabilities $0.3, 0.2, \dots, 0.1$, and other source of information tell you $c_3, c_4, \dots, c_k$ are impossible (i.e. have zero probability), then yes, you should prefer $c_1$ over $c_2$, since the predicted probabilities tell you something about relative ordering of the classes. You can find formal argument for this in here.

Answer 2

I haven't done any naive Bayes work, but the goal of classifying subjects also occurs in latent class analysis. I will focus on this part of the question:

Are the probabilities meaningful and interpretable in any way?

Yes. Yes, they are. They tell you how uncertain you are about the observation's classification. If your vector of class probabilities looks like (1, 0, 0, ... 0), you're obviously very certain that the observation has to be in class 1. If you have equally sized 10 classes and your vector looks like (0.1, 0.1, 0.1, ... 0.1), then you clearly have no idea how to classify your observation on whatever existing dimensions you have. It may be worth reading my answer on a different question.

This would imply a classification to Class 3 without any additional knowledge. Now I get an additional piece of information which tells me that the instance to classify belongs either to Class 1 OR to Class 2...Is it possible to assign the instance to Class 1 because I know that it holds the highest probability among the remaining classes if my information is right?

I am not exactly clear what you're asking. You're describing a situation where the model class was class 3 (i.e. most likely to be in class 3, albeit with considerable uncertainty). You receive new information that makes the modal class ... what? Was it really that class 1 and 2 have equal probabilities after receiving this new information? If this is true, but you're really in a scenario where your decision rule is to assign to the modal probability, then it seems equally principled to leave the modal class unassigned, or to assign via virtual coin flip.

Most properly, though, you do have some uncertainty in which observations belong to which class. Concepts like entropy (or, seeing as you're an economist, the Herfindahl index) are relevant for helping you quantify how much uncertainty you have about those classifications.