I have implemented Maximum-entropy Markov model (MEMM) for the Named entity recognition (NER) problem. I have four classes: geographical, people, material (book titles etc) and other.
other is overrepresented in the training & test datasets. It occurs 88% (about 40k samples) of the time.
I'm sure other authors must have had this problem. How do you suggest I go about solving this problem?
EDIT: A single data sample is a whole tagged sentence. Most of the sentence is tagged as
other. I'm guessing the resampling techniques won't work here? Do you suggest I use single tokens for training dataset, instead of whole sentences?
EDIT 2: In response to my post being a duplicate: you do realize you've linked to a question that was asked 2 years after mine, right?