Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I try to train a neural network using a dataset with several classes $c_1, c_2, \dotsc, c_{10}$. The class $c_1$ has a lot more entries in the training set than the other classes, and this makes my neural network to classify most of the the test set entries as being $c_1$.
What preprocessing should I make?
You are coping with an imbalanced dataset. Lucky for you, you are not alone. This is a common problem.
For surveys on the topic see Editorial: Special Issue on Learning from Imbalanced Data Sets (6 pages) and Learning from Imbalanced Data (22 pages)
The method I like best is the following: The method is based of the boosting algorithm Robert E. Schapire presented at "The strength of weak learnability" (Machine Learning, 5(2):197–227, 1990. The Strength of Weak Learnability ).
In this paper, Schapire presented a boosting algorithm based on combining triplets of 3 weak learners recursively. By the way, this was the first boosting algorithm.
We can use the first step of the algorithm (even without the recursion) to cope with the lack of balance.
The algorithm trains the first learner, L1, one the original data set. The second learner, L2, is trained on a set on which L1 has 50% chance to be correct (by sampling from the original distribution). The third learner, L3, is trained on the cases on which L1 and L2 disagree. As output, return the majority of the classifiers. See the paper to see why it improves the classification.
Now, for the application of the method of an imbalanced set: Assume the concept is binary and the majority of the samples are classified as true.
Let L1 return always true. L2 is being trained were L1 has 50% chance to be right. Since L1 is just true, L2 is being trained on a balanced data set. L3 is being trained when L1 and L2 disagree, that is, when L2 predicts false. The ensemble predicts by majority vote; hence, it predicts false only when both L2 and L3 predict false.
I used this method in practice many times, and it is very useful. It also has a theoretical justification so all fronts are covered.
flip coin; if heads, return the first instance $v$ from the original distribution for which the predicted label is correct; else return the first instance $v$ from the original distribution for which the predicted label is incorrect. I am confused how this would return a set of points, as each call gives one point.
$\endgroup$
An important point to make here is that classifying all of the patterns as $c_1$ might be the correct answer, from a statistical decision theory perspective. If misclassification costs are equal, we want to classify the pattern according to the class with the highest posterior probability, i.e. $p(C = c_i|\vec{x})$, which depends on the prior probabilities. So it may be that the prior probabilities of the minority classes are sufficiently low, and the distribution of patterns for each class sufficiently broad, that the true posterior probability is always highest for the majority class. If this is not acceptable for your application, then that probably means that your misclassification costs are not equal, so you need to think about what the costs of each kind of misclassification actually are, and build that into your cost function used to train the network.
Note if your network outputs estimates of probability of class membership, then it is offten possible to post-process the output of the trained network, rather than pre-process and retrain - see the excellent boook by Chris Bishop ("Neural Networks for Pattern Recognition").
In my experience, it isn't so much a "class imbalance problem" per se, but that there are just too few patterns from the minority class to accurately estimate their distribution, and if you increase the size of the dataset (but keep the ratios the same) the problem often goes away.
