2
$\begingroup$

Can someone explain what problems occur mathematically when providing a highly imbalanced training data set to a Naive Bayes Classifier?

Whats the mathematical reason for a resulting poor performance due to the imbalanced training data?

Let's say a training data set of 99 samples of negative label and 1 sample of positive label.

$\endgroup$

1 Answer 1

2
$\begingroup$

Naive Bayes theorem assumes independence and used Bayes theorem to calculate the probabilities

$$ p(C_k, x_1, x_2, \dots, x_m) = p(C_k) \, \prod_{j=1}^m p(x_j \mid C_k) $$

where we use the empirical estimates of the probabilities, e.g.

$$ p(C_k) = \frac{\text{the number of samples with class }C_k}{\text{total number of samples}} $$

So if for some class you have small number of samples, then the estimates for this class would be imprecise. Imagine that your data is the weather (sunny vs cloudy) and you want to predict if Joe is going to eat an ice cream on a given day. The data you have is shown below.

weather | ice cream
--------|----------
sunny   | no
cloudy  | yes
sunny   | yes
sunny   | yes
sunny   | yes

Now if you were going to predict what is the probability that Joe will eat ice cream on cloudy weather, it would be $p(x|y) \, p(y) = 0/1 \times 4/5 = 0$ just because you had only one sample, so your algorithm assumes anything else then it saw as "impossible". To correct for those, we would usually use Laplace smoothing and replace $0$ with some small, arbitrary value, so instead of zeroes we would end up with some numbers, but they still would not give us a precise result. This is an extreme case, but the general problem is that with small samples you get imprecise estimates.

The problem is not imbalanced data, but small number of samples. With imbalanced data, the algorithm gives bad predictions for the smaller class, but if you had little samples for both classes, it would be equally bad for both. Unbalanced data is generally not a problem, but insufficient data is.

Another problem may be is the proportion of class $C_k$ in your data differs from the proportion in the population (say you have 20% of females in your data, while in the population the fraction is closer to 50%), but then, to fix for that, just replace $p(C_k)$ with the true proportions (i.e. 0.5 rather then 0.2 in this example). This is possible because the conditional probabilities $p(x_j|C_k)$ are in each case calculated within the classes (to calculate $p(x_j|C_k)$ you count how often $x_j$ appeared for samples with $C_k$ class), while the "correction" for the class size is done by multiplying with $p(C_k)$ (check nice, worked example of how Bayes theorem works for more details). The class size does not enter the equation otherwise.

Moreover, you need to remember that naive Bayes gives you imprecise estimates of probability by design, so they would never be precise.

$\endgroup$
5
  • $\begingroup$ Hey Tim, many thanks for your comprehensive response but as you said you are describing the problem of "insufficient data". I was looking for reasons why NB could suffer from imbalanced data set. Do you think NB doesn't suffer from just imbalanced class distributions at all? Hence, it would not make sense to apply random over-sampling whereas SMOTE-sampling could help due to additional samples tacking "insufficient data". $\endgroup$ May 27, 2020 at 14:19
  • $\begingroup$ @SamSampleman as said above, the problem might be not enough data, but oversampling does not solve it, because it would only repeat some samples, so you don't gain new information by this. If the problem is that the distributions differ in your data vs population, then you only need to use the population $p(C_k)$ probabilities, instead of those in your data to correct it, as this is the only place where the relative sizes of the classes are considered by the algorithm. $\endgroup$
    – Tim
    May 27, 2020 at 14:40
  • $\begingroup$ Okay, apparently my question is still not quite clear. I'm totally aware about the "insufficient data" problem you have explained comprehensively. Let me ask you an additional question and please do not refer to "insufficient data" or to different distributions among data and population. Does NB suffer from highly imbalanced class distribution at all (assume same imbalance within the sample and population)? Consider problems of other algorithms when applied on imbalanced training data (e.g. neural networks). $\endgroup$ May 28, 2020 at 8:26
  • $\begingroup$ @SamSampleman the only problems, and solutions, are mentioned above. Also keep in mind that imbalanced data is a non-issue in many cases, there's no "imbalanced data problem" in general, just in some cases some additional care may be needed. $\endgroup$
    – Tim
    May 28, 2020 at 8:30
  • $\begingroup$ @SamSampleman see also my edit. $\endgroup$
    – Tim
    May 28, 2020 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.