How to deal with a highly unbalanced classification problem? I currently have data where there are 5 labels, 1,2,3,4,5 for my $Y$ variable and a set of associated predictors $X$. 
The problem is, I have around $10000$ observations with label $1$ and around only $100$ for label 2, $30$ for label 3, and finally down to $10$ for label 5. I am wondering what is a good way to calculate the expected value to predict for a label. Would it be wise to compute probabilities for each class label, then apply the probabilities into a weighted sum? What would someone do to handle this?
 A: Oversampling
One approach would be to use oversampling, which aims to generate artificial instances of minority classes based on the data available.
Adjusted Cost Function
The class imbalance impacts the training and evaluation of a classification model. In your case, using uniform costs for each type of misclassification might result in the classifier simply predicting every candidate instance as being of the majority class. This will score well if you're using cost insensitive evaluation measures like accuracy, but in fact, the patterns associated with the minority classes will not be considered by your model.
By defining a cost matrix for your problem and using it in the cost function of a classifier (for example in an SVM or with kNN) can help mitigate the negative effects of the imbalance. In this answer to a question on the topic I've explained how misclassification cost with a cost matrix can be used in evaluating a model.
To the research!
Imbalanced data is a topic that is subject to quite a bit of research in the field. I've heard some interesting talks about the topic during a workshop this year, maybe you can find interesting material in the proceedings. Here you might find other, more recent, approaches.
