I'm working on a classification task, where the target can be one of many (30k+) categories, and I know some of these categories are much closer to each other than the others (but I don't know exactly which ones).
It seems the cross entropy loss may not be the best choice because it treats every category independently.
For instance say there're five almost indistinguishable categories, and my prediction is 0.2 for each and zero for others, which I think is a very good prediction, but it still gets a large cross entropy loss. I'm thinking maybe I can modify the loss to something like "if the predicted probability is above 0.2 for the ground truth category, it causes no loss", will that work?
What should be a proper loss in my case?
or what else can I do (to impose the assumption that some categories are similar)?
One thing I thought of is to just set the loss of top K predicted categories to zero. I think it has similar effect as early stopping, since they both don't encourage the model to be too confident of their predictions (overfit) as opposed to a cross entropy loss.
As January has suggested, I think it makes sense to apply a hierarchical clustering method first, then the "degree of difference" between categories can be weighted according to the hierarchy.
Not the same as Principled way of collapsing categorical variables with many levels? The linked question is about using categorical attributes as inputs, not targets.