Bernoulli Naive Bayes with Unbalanced Classes of Binary Features I'm using a multivariate Bernoulli model with a Naive Bayes classifier on binary features. It's giving a lot of very low estimates of the probability that each instance is correctly classified. I suspect that this may be because the classes are very unbalanced.
What would be the best over sampling technique that I could try to rebalance my multivariate binary data please?
I thought that SMOTE (Synthetic Minority Oversampling TEchnique) might be useful. This synthesises elements for the smaller classes, based on those that already exist. It randomly picks a point from the smaller classes then it adds synthetic points between it and its neighbors. This seemed good as I thought that it would not cause so much over fitting as resampling the smaller classes might. It however doesn't work on binomial data. Is there a binomial equivalent please?
 A: The fact that your algorithm is giving very low probabilities when asked to predict whether an unlabelled example belongs to a given class, should not necessarily be seen as a bad thing. Over- and under-sampling will mean the algorithm assigns higher probabilities..but these are incorrect.
Consider the following, you have a data set of medical records, and you want to use a number of features to predict whether an individual has a rather rare disease. The classes are distributed such that only 1/10000 people in the data set has this rare disease. What machine learning will likely let you do, is find people whose features make them much more likely than a randomly selected member of the population to have the disease, but that will probably not mean you can determine with high confidence whether they have it. In this case, because the base rate is 1/10000 or 0.01%, success might look like being able to say about certain individuals, that they have a 1/100 or 1% chance of having the disease (and thus 100x more likely than a randomly selected member of the population). If you found out that you had a 1% chance of being ill, when the base rate in the general population is 0.01%, you'd probably want to get yourself checked out. Furthermore, if we only scanned people who had a 1% chance or greater, we'd get a 100x higher a hit rate than if we scanned everyone, which might be a better use of resources. 
In order for an algorithm to be able to find people who have a >50% chance of being ill, when the base rate is 0.01%, you would need for there to be very strong signal in the data indeed. As a rule of thumb, you care about how much more certain you can be that somebody is in the minority class, than you would be of a randomly selected (or selected according to the current best procedure) individual. That usually roughly corresponds to how impactful your algorithm will be in practice.
If you up/downsample, your algorithm might start telling you that there's a 50% chance you have a serious disease, when the true probability is 1%, does that desirable? 
A: The simplest over sampling technique is to repeat the smaller classes. Here is a Python script to do that.
I will try to add some noise to it. This is because over sampled data can be very repetitive and this causes machine learning tools to over fit to the over sampled data. I would use SMOTE to add the noise but I don't think that works for binary. I'll see if I can swap some values within classes.
A colleague commented: "I don't think oversampling is a good idea... If you have very many classes and insufficiently correlated attributes then I'm afraid there's not that much you can do to improve the accuracy. As a first step, I'd try to pool some of the classes (if that's possible, I don't know if that would work in your application)."
