# Classification with a neural network when one class has disproportionately many entries

I try to train a neural network using a dataset with several classes $c_1, c_2, \dotsc, c_{10}$. The class $c_1$ has a lot more entries in the training set than the other classes, and this makes my neural network to classify most of the the test set entries as being $c_1$.

What preprocessing should I make?

You are coping with an imbalanced dataset. Lucky for you, you are not alone. This is a common problem.

For surveys on the topic see Editorial: Special Issue on Learning from Imbalanced Data Sets (6 pages) and Learning from Imbalanced Data (22 pages)

The method I like best is the following: The method is based of the boosting algorithm Robert E. Schapire presented at "The strength of weak learnability" (Machine Learning, 5(2):197–227, 1990. The Strength of Weak Learnability ).

In this paper, Schapire presented a boosting algorithm based on combining triplets of 3 weak learners recursively. By the way, this was the first boosting algorithm.

We can use the first step of the algorithm (even without the recursion) to cope with the lack of balance.

The algorithm trains the first learner, L1, one the original data set. The second learner, L2, is trained on a set on which L1 has 50% chance to be correct (by sampling from the original distribution). The third learner, L3, is trained on the cases on which L1 and L2 disagree. As output, return the majority of the classifiers. See the paper to see why it improves the classification.

Now, for the application of the method of an imbalanced set: Assume the concept is binary and the majority of the samples are classified as true.

Let L1 return always true. L2 is being trained were L1 has 50% chance to be right. Since L1 is just true, L2 is being trained on a balanced data set. L3 is being trained when L1 and L2 disagree, that is, when L2 predicts false. The ensemble predicts by majority vote; hence, it predicts false only when both L2 and L3 predict false.

I used this method in practice many times, and it is very useful. It also has a theoretical justification so all fronts are covered.

• the first two links seem to go to the same paper. Feb 4, 2016 at 7:50
• That's because it is a good paper... I fixed the link, Thanks.
– DaL
Feb 4, 2016 at 8:15
• @DanLevin: Would you mind explaining how to construct the "set on which L1 has 50% chance to be correct (by sampling from the original distribution)?" I am looking at the original paper by Schapire (thanks, by the way, for sharing it) where he defines these "oracles" as follows: flip coin; if heads, return the first instance $v$ from the original distribution for which the predicted label is correct; else return the first instance $v$ from the original distribution for which the predicted label is incorrect. I am confused how this would return a set of points, as each call gives one point. Jan 2, 2017 at 22:04
• The use of oracles is due to the mathematical framing. I practice what will you do is to label all the samples using L1. Now you will have a set on which L1 is right and a set on which L1 is wrong. Sample x/2 items from each one of them and you are done.
– DaL
Jan 3, 2017 at 7:11
• What if I use probabilities instead of classes? For example, my L1 and L2 learners return something like [0.15 0.85] instead of 1. How can I use L3 learner to improve my score (I use logloss) in such case? Feb 26, 2017 at 19:12

An important point to make here is that classifying all of the patterns as $c_1$ might be the correct answer, from a statistical decision theory perspective. If misclassification costs are equal, we want to classify the pattern according to the class with the highest posterior probability, i.e. $p(C = c_i|\vec{x})$, which depends on the prior probabilities. So it may be that the prior probabilities of the minority classes are sufficiently low, and the distribution of patterns for each class sufficiently broad, that the true posterior probability is always highest for the majority class. If this is not acceptable for your application, then that probably means that your misclassification costs are not equal, so you need to think about what the costs of each kind of misclassification actually are, and build that into your cost function used to train the network.

Note if your network outputs estimates of probability of class membership, then it is offten possible to post-process the output of the trained network, rather than pre-process and retrain - see the excellent boook by Chris Bishop ("Neural Networks for Pattern Recognition").

In my experience, it isn't so much a "class imbalance problem" per se, but that there are just too few patterns from the minority class to accurately estimate their distribution, and if you increase the size of the dataset (but keep the ratios the same) the problem often goes away.