Named entity recognition and class imbalance I have implemented Maximum-entropy Markov model (MEMM) for the Named entity recognition (NER) problem.
I have four classes: geographical, people, material (book titles etc) and other.
Class other is overrepresented in the training & test datasets. It occurs 88% (about 40k samples) of the time.
I'm sure other authors must have had this problem. How do you suggest I go about solving this problem?
EDIT: A single data sample is a whole tagged sentence. Most of the sentence is tagged as other. I'm guessing the resampling techniques won't work here? Do you suggest I use single tokens for training dataset, instead of whole sentences?
EDIT 2: In response to my post being a duplicate: you do realize you've linked to a question that was asked 2 years after mine, right?
 A: Some things you can try:


*

*Oversample your target classes. Insert duplicate records of your other three classes to augment your training dataset

*Undersample the negative responses. Instead of including all instances of other in your training data, only use a small portion.

*Bootstrap undersample the negative responses. This is probably your most robust option of those I'm presenting. Start by seeding your training data with the non-other classified records. Then train each bootstrap iteration augmenting the seed training set with a different random sample (selected with replacement) from the other class. You can then either derive a confidence interval for your models classifications from the bootstrapping procedure (as Kaushik suggested), or treat the models you generated as an ensemble and combine their scores using an average or majority vote to determine your classifications. You can even implement boosting here if you want.
A: One method that I have used with success is resampling the data. I run bootstraps by taking N samples from each class where N is the size of the class with the smallest samples. The N samples are chosen randomly without replacement. Then I split each resampled class into a training and test set (say 70-30 split) and run my classifier. For each boot strap I get a score. I run about 1000 bootstraps to get confidence intervals on my score.
The resampling forces each class to yield the same number of train and test samples to get around the class imbalance problem, but then I do bootstraps to get a meaningful mean score and a confidence interval.
For what it's worth, I have a short post with some simple Python code discussing imbalanced classes.
