Consider the following simplified sample dataset representative of the complex dataset. The alphabets a, b, c .. represent any word in English vocabulary. X1, X2 and Y1 are the actual classes they should be assigned which have been broken down into hierarchies as shown in the table below. It is evident from the sample dataset that while 1 and 2 occur as sub-classes X, class 2 does not occur as a sub-class of class Y. But how do we make a hierarchical classifier replicate such behaviour where the sample space of classes ({X1, X2, Y1, Y2} in the example) is much bigger than the event space ({X1, X2, Y1} in the example).

| Text       |  Class          |  Sub Class      |
| a b c d    |  X              |  1              |
| b d e a f  |  X              |  2              |
| f g a      |  Y              |  1              |

To give an idea about the actual complexity of the data, here are a few details. The number of such texts is around 100,000 which belong to 100,000 classes that are hierarchically labeled using alphanumeric class labels. The number of hierarchies is around 5 with level 1 having around 25 distinct classes, level 2 with 100 distinct classes, level 3 with 20 distinct classes, level 4 with 15 distinct classes and level 5 as 10 distinct classes, giving a total potential sample space of classes of 7.5 million.

One thought process in resolving this issue went about creating dummy data for the missing labels so that classifier does not predict a class from the sample space that is not present in the training data.

Any leads or edit suggestions in case the statements are not clear would be helpful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.