I'm dealing with the following problem:
I have a multiclass variable, y, with let's say 7 classes. The 7 classes are not evenly distributed, some are way more likely to occur than others. Let's say class A occurs in 40% of the cases, and the other classes all occur in 10% of the cases.
I'm trying to predict these classes with the variables in the matrix X. Given that it's rather hard to predict y with the given variables in X (accuracy of about 50%), some out-of-sample distributions tend to be quite close to the prior distribution (40%, 10%, etc.). Hence class A has a larger distribution by default.
When I move to predictions based on the largest probability, class A is chosen more often than expected (+/- 70%) and the other classes sometimes hardly occur, because by default the probabilities are smaller.
What can I do to improve my model? I was thinking about:
- Weight the probabilities, based on the frequency of the classes
- Choose my predictions in a different manner
- Train my model on a different set, such that all classes occur with the same frequency.
Example: if for observation k the probability for class B is 39%, and for class A 40%, I would prefer to assign this observation to class B, even though the probability for class A is somewhat larger.
Any advice, inspiration or "best practice" is more than welcome!
I'm using models like xgBoost/random forest, with multiclass probabilities. Output looks like:
A, B, C, D, E, F, G 0.4, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1