Mathematical implications of Naive Bayes Classifier on imbalanced data set

Question

Can someone explain what problems occur mathematically when providing a highly imbalanced training data set to a Naive Bayes Classifier?

Whats the mathematical reason for a resulting poor performance due to the imbalanced training data?

Let's say a training data set of 99 samples of negative label and 1 sample of positive label.

Answer 1

Naive Bayes theorem assumes independence and used Bayes theorem to calculate the probabilities

$$ p(C_k, x_1, x_2, \dots, x_m) = p(C_k) \, \prod_{j=1}^m p(x_j \mid C_k) $$

where we use the empirical estimates of the probabilities, e.g.

$$ p(C_k) = \frac{\text{the number of samples with class }C_k}{\text{total number of samples}} $$

So if for some class you have small number of samples, then the estimates for this class would be imprecise. Imagine that your data is the weather (sunny vs cloudy) and you want to predict if Joe is going to eat an ice cream on a given day. The data you have is shown below.

weather | ice cream
--------|----------
sunny   | no
cloudy  | yes
sunny   | yes
sunny   | yes
sunny   | yes

Now if you were going to predict what is the probability that Joe will eat ice cream on cloudy weather, it would be $p(x|y) \, p(y) = 0/1 \times 4/5 = 0$ just because you had only one sample, so your algorithm assumes anything else then it saw as "impossible". To correct for those, we would usually use Laplace smoothing and replace $0$ with some small, arbitrary value, so instead of zeroes we would end up with some numbers, but they still would not give us a precise result. This is an extreme case, but the general problem is that with small samples you get imprecise estimates.

The problem is not imbalanced data, but small number of samples. With imbalanced data, the algorithm gives bad predictions for the smaller class, but if you had little samples for both classes, it would be equally bad for both. Unbalanced data is generally not a problem, but insufficient data is.

Another problem may be is the proportion of class $C_k$ in your data differs from the proportion in the population (say you have 20% of females in your data, while in the population the fraction is closer to 50%), but then, to fix for that, just replace $p(C_k)$ with the true proportions (i.e. 0.5 rather then 0.2 in this example). This is possible because the conditional probabilities $p(x_j|C_k)$ are in each case calculated within the classes (to calculate $p(x_j|C_k)$ you count how often $x_j$ appeared for samples with $C_k$ class), while the "correction" for the class size is done by multiplying with $p(C_k)$ (check nice, worked example of how Bayes theorem works for more details). The class size does not enter the equation otherwise.

Moreover, you need to remember that naive Bayes gives you imprecise estimates of probability by design, so they would never be precise.