I've always wondered why no one using smoothing techniques for Naive Bayes which implicitly test which conditional probabilities should actually be used and which ones shouldn't.

The formula for Naive Bayes is as follows $$P(Y=y_i|X=x_i)= \frac{P(X=x_k|Y=y_i) P(Y=y_i)} {\sum_j P(X=x_k|Y=y_i) P(Y=y_j)}$$

My question is: how do we know that using $P(X=x_k|Y=y_i)$ isn't just adding noise? For example, if $P(X=x_k|Y=y_i)=1$, but there is only one instance of $P(Y=y_i)$, then we don't exactly have high confidence that the two are conditionally dependent- we just have one case. Laplace smoothing seems to remedy this in a slightly haphazard way (adds 1 to each bin, so it will reduce $P(X=x_k|Y=y_i)$ by some amount- but is this the right amount?).

What if instead, we were to test whether $P(X=x_k|Y=y_i)=P(Y=y_i)$ using a one sample proportion test at some confidence $\alpha$, i.e. test whether x and y are conditionally dependent for these two classes. If we are not confident that these two are dependent, then we can use: $$P(Y=y_i|X=x_i)= \frac{P(Y=y_i)} {\sum_j P(X=x_k|Y=y_i) P(Y=y_j)}$$ and essentially discount the conditionality between these two classes (here obviously the result is useless, but we usually do this across many variables so discounting one variable may be useful).

If they are dependent, we can still adjust $P(X=x_k|Y=y_i)$ by taking the bound of it's confidence interval closes to $P(X=x_k)$ essentially saying- "we are $\alpha$% confident that at least this level of dependence exists.

Does something like this already exists? If no, is there a mathematical issue with the proposed idea?


1 Answer 1


Since no one else has chimed in, here my preliminary, not super well developed thoughts on this.

if $A$ and $B$ are independent, then $P(A|B)P(B) = P(A)P(B)$. Therefore, if we can't find are unable to reject the hypothesis that $A \perp B$, then we should expect that replacing the likelihood term $P(A|B)$ (let's call this the "conditional likelihood") with $P(A)$ (let's call this the "independent likelihood") shouldn't really have much if any effect on the model, since the two quantities will be extremely similar. Also, and probably more importantly, if you fix the numerator like you're suggesting (such that it's not longer conditional on the data), your denominator is already fixed so you're essentially left with predicting all observations the same: $P(B|A) = P(B)$. In a regression context, this would be equivalent to a model whose only non-zero term is the intercept, i.e. $E[Y]$.

The effect of replacing the conditional likelihood with the independent likelihood might be more pronounced when $N$ is small, but in general I think the exchange you suggested shouldn't really make any difference.

Also, I think this is sort of a weird step to take philosophically. Presumably, you're including the variable $A$ in your model because you believe it to be predictive of $B$, i.e. that there is a conditional relationship.

In the multiclass case, I suppose situations could arise where a variable $V_i$ is not correlated with class $C_i$, but this variable is still correlated with other classes and this class with other variables, such that $V_i \perp C_i, \exists j,k:V_i|C_j\&V_k|C_i$ where $i \ne j,k$. In such a situation, I suppose you are not subject to the philosophical problem, but I'd expect that swapping the conditional likelihood for the independent likelihood shouldn't really have any effect, as described earlier.

Ultimately, the answer is probably if you're using a naive bayes model and you have a feature that is essentially uncorrelated with your target, you should probably just remove that feature from your model since it isn't adding any information. Keeping it in, especially if you used your idea, would basically be equivalent to multiplying all of your probabilities by a scalar, which wouldn't change their relative rankings.

  • $\begingroup$ "The effect of replacing the conditional likelihood with the independent likelihood might be more pronounced when N is small" This is the exact case that we are looking to address. If N is small, the conditional likelihood is not reliable and we want to adjust it. $\endgroup$
    – ALutes
    Nov 6, 2015 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.