Hierarchical decomposition of an imbalanced multiclass classification problem

Question

I have a heavily imbalanced multiclass text classification problem: one class is very probable a priori (P), while the remaining four ones are about equally improbable (I1 to I4). I have performed many experiments using a single multiclass Random Forest, trying to model all classes at once (with the goal of optimizing the negative loglikelihood).

The heavily imbalanced structure of that particular problem made me think however that a better strategy would be to decompose it in a hiercharchy, by conflating the improbable classes into one (I): a first-level RF could be used to model the P vs I binary problem, while a second-level RF could focus on the specialized four-class problem. It's then easy to combine the outputs of the two models into one.

My intuition was that this decomposition should work well, because each model has a simpler task than the overall one, with more training information (i.e. being less diluted in a wider and unevener set of classes). But even though I've been very careful to make my two implementations very similar and comparable, I found that the single multiclass model clearly outperforms the hierachical one, contrary to my intuition. I'd like to know if there's a way to understand this result.

Answer 1

Hierarchical classification models frequently fail for different reasons. This is why flat classification methods based on one-vs-rest are usually preferred.

One of the main reasons discussed in the literature is that once an error is made in the upper levels of the hierarchy the model has no way to recover. To analyze this type of problem in your case you should calculate the error in the first step P vs I binary problem. This will be highly indicative as if the accuracy of your model is low there, it will be even lower in the last level.

Following that, the design of the hierarchy is also an issue. Things that are intuitive for humans do not necessarily yield the best performance. In your case for instance, it may be intuitive to split the problem as you describe (P vs I). From a machine learning perspective however, this may be sub-optimal due the the specificity of your data etc.. To better understand this, imagine that P vs I is difficult (not many examples of the I class, similar features...). There may be though another problem like I1 Vs Others that is simpler and better suited for the root of your hierarchy. This is a serious issue to be considered when designing your hierarchy. You simply need to start from the simpler (easier) problem (top of the hierarchy) and put more difficult problems to the bottom because you can not overcome from errors.

By the way, there are several, arguably simpler ways to deal with imbalanced datasets like adding weights to the infrequent classes, subsampling the frequent class, oversampling the infrequent and others. You may want to try them first.