12
$\begingroup$

I have implemented Maximum-entropy Markov model (MEMM) for the Named entity recognition (NER) problem. I have four classes: geographical, people, material (book titles etc) and other.

Class other is overrepresented in the training & test datasets. It occurs 88% (about 40k samples) of the time.

I'm sure other authors must have had this problem. How do you suggest I go about solving this problem?

EDIT: A single data sample is a whole tagged sentence. Most of the sentence is tagged as other. I'm guessing the resampling techniques won't work here? Do you suggest I use single tokens for training dataset, instead of whole sentences?

EDIT 2: In response to my post being a duplicate: you do realize you've linked to a question that was asked 2 years after mine, right?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
6
+100
$\begingroup$

Some things you can try:

  1. Oversample your target classes. Insert duplicate records of your other three classes to augment your training dataset

  2. Undersample the negative responses. Instead of including all instances of other in your training data, only use a small portion.

  3. Bootstrap undersample the negative responses. This is probably your most robust option of those I'm presenting. Start by seeding your training data with the non-other classified records. Then train each bootstrap iteration augmenting the seed training set with a different random sample (selected with replacement) from the other class. You can then either derive a confidence interval for your models classifications from the bootstrapping procedure (as Kaushik suggested), or treat the models you generated as an ensemble and combine their scores using an average or majority vote to determine your classifications. You can even implement boosting here if you want.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ could you please introduce me some references related to undersampling for NER ? $\endgroup$
    – user1844172
    Commented Aug 13, 2014 at 7:29
2
$\begingroup$

One method that I have used with success is resampling the data. I run bootstraps by taking N samples from each class where N is the size of the class with the smallest samples. The N samples are chosen randomly without replacement. Then I split each resampled class into a training and test set (say 70-30 split) and run my classifier. For each boot strap I get a score. I run about 1000 bootstraps to get confidence intervals on my score.

The resampling forces each class to yield the same number of train and test samples to get around the class imbalance problem, but then I do bootstraps to get a meaningful mean score and a confidence interval.

For what it's worth, I have a short post with some simple Python code discussing imbalanced classes.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I use whole tagged sentences as data samples, where most of the sentence is tagged as other. Will resampling work here as well? $\endgroup$
    – Howie
    Commented Jul 17, 2013 at 6:42
  • 1
    $\begingroup$ I imagine each word (or phrase) is a sample. The re-sampling should be done based on the samples (tagged words/phrases). It sounds to me like the sentence is just a container for a variable number of samples. $\endgroup$
    – Kaushik Ghose
    Commented Jul 17, 2013 at 12:33
  • 1
    $\begingroup$ @Howie the bulk of a single sentence may be tagged other, but do all of the sentences in your training set include a named entity? If not, I'd recommend creating a binary flag associated with each sentence in your training data to indicate whether or not it includes a names entity and apply the over/under sampling to whole sentences to amplify the proportion of sentences containing named entities in your training data. $\endgroup$
    – David Marx
    Commented Jul 17, 2013 at 16:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.