Consider a classification task wherein the training data has around 100,000 sentences with around 1000 labels. These 100,000 sentences can possibly be grouped hierarchically. The task at hand is given a random input string, find out the most similar of the 100,000 sentences and assign the corresponding label to the random input string.
The random input string may be only semantically related to one or more of the 100,000 labelled sentences and hence can be assigned one or more of the 1000 labels.
Also in the training data, the number of sentences of belonging to individual classes are highly imbalanced.
I have considered a lot of different ways but am not sure which would be the best way to go about it. The literature says that in general one should convert the problem to reduce the number of classes by following hierarchical classification. But has the drawback that the error rate cascades over the levels of hierarchy and results in poor overall performance
Other popular suggestion is convert the problem to one vs one similarity problem but this approach presents the issue of high class imbalance.
Any ideas about how to tackle this problem ?