I'm working on a classification task, where the target can be one of many (30k+) categories, and I know some of these categories are much closer to each other than the others (but I don't know exactly which ones).

It seems the cross entropy loss may not be the best choice because it treats every category independently.

For instance say there're five almost indistinguishable categories, and my prediction is 0.2 for each and zero for others, which I think is a very good prediction, but it still gets a large cross entropy loss. I'm thinking maybe I can modify the loss to something like "if the predicted probability is above 0.2 for the ground truth category, it causes no loss", will that work?

What should be a proper loss in my case?
or what else can I do (to impose the assumption that some categories are similar)?

One thing I thought of is to just set the loss of top K predicted categories to zero. I think it has similar effect as early stopping, since they both don't encourage the model to be too confident of their predictions (overfit) as opposed to a cross entropy loss.

As January has suggested, I think it makes sense to apply a hierarchical clustering method first, then the "degree of difference" between categories can be weighted according to the hierarchy.

Not the same as Principled way of collapsing categorical variables with many levels? The linked question is about using categorical attributes as inputs, not targets.

  • $\begingroup$ Are you willing to go towards a multi-label classification? You could then create a "mother" label regrouping the 5 almost indistinguishable categories. Say you have at the moment the categories : C1, C2, C3, C4 and C5 which are really close you could tag your input as both: C (the "mother label") and C2. This way even if that input was in reallity a C4 you will still be partialy right in your prediction. Just an idea i got, don't know if this actually solve your problem. $\endgroup$ – LoulouChameau Aug 23 '16 at 14:45
  • $\begingroup$ @LoulouChameau thank you, i'm not sure if i'm able to do that, as i only have one label per sample and i dont know exactly which categories are close. $\endgroup$ – dontloo Aug 23 '16 at 14:51
  • 2
    $\begingroup$ Ah ok i see, well maybe you could put your data through a clustering algorithm and look by yourself what kind of input get grouped together by the clustering. That will give you an idea of which categories are close and which are not. But that's really quite a stretch I have to admit. Good luck in your work anyway. $\endgroup$ – LoulouChameau Aug 23 '16 at 14:55
  • $\begingroup$ @LoulouChameau Thank you, and it's a bit hard to get meaningful clusters as well as the data are images. $\endgroup$ – dontloo Aug 24 '16 at 1:39

Here is a possible solution based on my experience in high throughput biology.

Before you launch your classification machinery, you need to learn more about the subject of your studies. I see two options.

  1. Use external knowledge. What are these labels? How can they be related? Do you have an other source of information that would group them together?

  2. Lacking external knowledge, I would sacrifice a randomly selected fraction F of your data to understand which labels cluster together. Build the clusters based on the set F, using cross-validation and whatever clustering validation schemes you think are worthwhile. Then, re-label the remaining data (F') using the new clusters as labels, which should optimally reduce the number of unnecessary categories. However, the split into F and F' is crucial: if you build your new categories using the full data set in which you then run your machine learning, you will end up overfitting.

  • $\begingroup$ Hi thanks for answering. I don't think there's any unnecessary categories in my case, every category is independent of each other. Maybe I could try a hierarchical clustering method. $\endgroup$ – dontloo Dec 17 '18 at 12:05
  • $\begingroup$ Then I do not understand what you mean by "some categories are much closer to each other" $\endgroup$ – January Dec 17 '18 at 12:13
  • 1
    $\begingroup$ Maybe you should tell us more about the data you are working with. $\endgroup$ – January Dec 17 '18 at 12:15
  • 2
    $\begingroup$ It's actually face images from thousands of people, where some people do look very similar $\endgroup$ – dontloo Dec 17 '18 at 12:17
  • $\begingroup$ So some people are very similar and you cant tell their photographs apart, is that what you are saying? $\endgroup$ – January Dec 17 '18 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.