I came across a kind of classification problem.

Suppose I have a dataset with labels, where each label is one of a1, a2, a3, b1, and b2. I want to make a classifier that estimates "coarse" label, a or b.

Here is a concrete example. Imagine a set of animal images labeled by the its species. We want make a classifier that predicts big classes, like mammal, reptile or fish.

Q1. Does anybody know the name of this kind of classification problem setting?

To solve the problem, we can think of several approaches.

  1. First make the label coarse (a1->a, a2->a, b1->b,... ) and solve it as a standard classification problem.

  2. First solve the standard classification problem, predicting a1, a2,... Then make the label coarse.

  3. First solve the standard classification problem, predicting a1, a2,... (level-0 classifier) Then use the output scores of level-0 classifier as input, and make a level-1 classifier which outputs a, b,.. (stacked generalization approach)

Q2. What kind of approach is good for this kind of problem?

I guess the approach 1 is not optimal since part or label information is thrown away. As Marc Claesen has pointeded out, this approach 1 is good if there is an example that is obviously a member of a but not obviously a member of one of a1, a2,...


I guess the approach 1 is not optimal since part or label information is thrown away.

Not necessarily. It depends on the data. Consider the following scenarios in which you would use opposite approaches.

If all subclasses per general class ($a_1$, $a_2$, $a_3$ for $a$) are very similar, it is likely easier to solve the problem of $a$ vs $b$ directly. Intuitively, there may be instances that clearly belong to $a$ but may not correspond as clearly to any of its subclasses.

In contrast, if the subclasses ($a_1$, $a_2$, $a_3$, ...) are dissimilar, it may not make sense to attempt to model the encompassing class $a$.

  • $\begingroup$ Thank your for your insightful comment. In some applications, coarse class comes from other contexts and may not necessarily similar in input space. If classification algorithm automatically recognize those data properties, that would help a lot. $\endgroup$ – ywat Dec 18 '13 at 4:24

You can use predictions (preferably real values, not binary) from model that uses all labels as input for prediction of final coarse labels.
Something similar to stacked generalization.

  • $\begingroup$ Thank you. I haven't know the notion of "stacked generalization." I have added your idea as the third approach. $\endgroup$ – ywat Dec 18 '13 at 4:09

I believe this is called hierarchical classification and has been studied. One approach is to use multi-task learning - i.e. use a decision tree or neural net in order to try and take the hierarchical information into account. For instance if you use a neural network, you could have your outputs be the fine class labels as well as the coarse labels. Then, as in mult-task learning, the latent features that will develop in the hidden layer will benefit both tasks - these features may not develop otherwise. This will only work though if there is some "common information" shared between the fine labels of a coarser class.

  • $\begingroup$ Thank you for your comment. I would like to clarify your suggestion. For neural network case, you mean to make a neural network with two outputs, sharing hidden layers? $\endgroup$ – ywat Dec 18 '13 at 13:26
  • $\begingroup$ No, if you had F fine classes and C coarse classes the network would hvae F+C classes. each training example would then have 2 labels. The goal would be to try and predict both the coarse and fine label correctly at the same time. In this way you might be able to learn both tasks better than if you trained these separately. A decision tree could also work. $\endgroup$ – user671931 Dec 18 '13 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.