5
$\begingroup$

I'm working on a classification task, where the target can be one of many (30k+) categories, and I know some of these categories are much closer to each other than the others (but I don't know exactly which ones).

It seems the cross entropy loss may not be the best choice because it treats every category independently.

For instance say there're five almost indistinguishable categories, and my prediction is 0.2 for each and zero for others, which I think is a very good prediction, but it still gets a large cross entropy loss. I'm thinking maybe I can modify the loss to something like "if the predicted probability is above 0.2 for the ground truth category, it causes no loss", will that work?

What should be a proper loss in my case?
or what else can I do (to impose the assumption that some categories are similar)?

Update
One thing I thought of is to just set the loss of top K predicted categories to zero. I think it has similar effect as early stopping, since they both don't encourage the model to be too confident of their predictions (overfit) as opposed to a cross entropy loss.

As January has suggested, I think it makes sense to apply a hierarchical clustering method first, then the "degree of difference" between categories can be weighted according to the hierarchy.

Update
Not the same as Principled way of collapsing categorical variables with many levels? The linked question is about using categorical attributes as inputs, not targets.

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you willing to go towards a multi-label classification? You could then create a "mother" label regrouping the 5 almost indistinguishable categories. Say you have at the moment the categories : C1, C2, C3, C4 and C5 which are really close you could tag your input as both: C (the "mother label") and C2. This way even if that input was in reallity a C4 you will still be partialy right in your prediction. Just an idea i got, don't know if this actually solve your problem. $\endgroup$
    – LoulouChameau
    Commented Aug 23, 2016 at 14:45
  • $\begingroup$ @LoulouChameau thank you, i'm not sure if i'm able to do that, as i only have one label per sample and i dont know exactly which categories are close. $\endgroup$
    – dontloo
    Commented Aug 23, 2016 at 14:51
  • 2
    $\begingroup$ Ah ok i see, well maybe you could put your data through a clustering algorithm and look by yourself what kind of input get grouped together by the clustering. That will give you an idea of which categories are close and which are not. But that's really quite a stretch I have to admit. Good luck in your work anyway. $\endgroup$
    – LoulouChameau
    Commented Aug 23, 2016 at 14:55
  • $\begingroup$ @LoulouChameau Thank you, and it's a bit hard to get meaningful clusters as well as the data are images. $\endgroup$
    – dontloo
    Commented Aug 24, 2016 at 1:39

1 Answer 1

Reset to default
3
$\begingroup$

Here is a possible solution based on my experience in high throughput biology.

Before you launch your classification machinery, you need to learn more about the subject of your studies. I see two options.

  1. Use external knowledge. What are these labels? How can they be related? Do you have an other source of information that would group them together?

  2. Lacking external knowledge, I would sacrifice a randomly selected fraction F of your data to understand which labels cluster together. Build the clusters based on the set F, using cross-validation and whatever clustering validation schemes you think are worthwhile. Then, re-label the remaining data (F') using the new clusters as labels, which should optimally reduce the number of unnecessary categories. However, the split into F and F' is crucial: if you build your new categories using the full data set in which you then run your machine learning, you will end up overfitting.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Hi thanks for answering. I don't think there's any unnecessary categories in my case, every category is independent of each other. Maybe I could try a hierarchical clustering method. $\endgroup$
    – dontloo
    Commented Dec 17, 2018 at 12:05
  • $\begingroup$ Then I do not understand what you mean by "some categories are much closer to each other" $\endgroup$
    – January
    Commented Dec 17, 2018 at 12:13
  • 1
    $\begingroup$ Maybe you should tell us more about the data you are working with. $\endgroup$
    – January
    Commented Dec 17, 2018 at 12:15
  • 2
    $\begingroup$ It's actually face images from thousands of people, where some people do look very similar $\endgroup$
    – dontloo
    Commented Dec 17, 2018 at 12:17
  • $\begingroup$ So some people are very similar and you cant tell their photographs apart, is that what you are saying? $\endgroup$
    – January
    Commented Dec 17, 2018 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.