Multi-class classification - confusing some classes is worse than others

Question

I am quite a Rookie concerning machine learning and now thinking about the following problem:

I want to assign a person to one out of 6 groups, let's say concerning their favorite colour --> works quite well with random forrest.

Now it is the case that categorizing a person from group 1 as group 3 (favorite colour is red but I predict green) is worse than categorizing this person as group 4 (favorite colour is red but I predict purple).

Is there an algorithm for this special use case? Or an option to "tune" my evaluation metrics so they consider the second best or "close to right" group, too?

Thanks a lot in advance! Sara

Answer 1

One thing you can do is define weights for your loss where confusing red with green has a higher weight than red with purple. You basically define something like a weight matrix and look up the weight for every given class and its prediction.

Research on image segmentation shows promising results with using approach (arXiv paper).

Answer 2

The problem is that categorization is a premature decision and makes it difficult to bring in the right utility/cost/loss function. Classification also invites you to make classifications when you shouldn't, e.g., when the probability of class membership is far from 0 or 1. If you estimate the probability of membership of all classes (using e.g. multinomial (polytomous) logistic regression) and come up with utility/cost/loss functions you can make optimum decisions. Your problem as stated would be well served by your translating the various classification error importances to a utility function to plug into the optimum Bayes decision that maximizes expected utility.