1
$\begingroup$

In a multi-class classification problem, models that I've trained (and which I intend to use in a majority voting ensemble) consistently get weak performance on a couple of specific classes. There is class imbalance, but the "bad" classes are not the least frequent classes. What strategies are there to boost performance specifically for those classes?

I had an idea but I'm not quite sure how to implement it: train an additional model that does binary classification on the weak class vs all others, then add that to the majority vote (either raw votes or probabilities) ensemble to give stronger weight to that class when other models get it wrong. Is this a reasonable approach?

$\endgroup$
2
  • $\begingroup$ Among their other mechanics, boosting methods like Adaboost increase the weight of misclassified points and decrease the weight of correctly classified points on each iteration. So, future classifiers in the ensemble concentrate more on the points misclassified so far. Although this works at the level of individual points rather than entire classes, it might do what you want if points from one class are systematically misclassified. But, it can increase sensitivity to outliers. You could try boosting your classifier, or think about how to borrow/modify this particular feature. $\endgroup$ – user20160 Jan 25 '17 at 20:53
  • $\begingroup$ @user20160 Thanks, I'll try an Adaboost method too. I've already tried xgboost and it was one of the better performers, but it still did poorly in the "bad" classes. $\endgroup$ – adatum Jan 25 '17 at 21:17
1
$\begingroup$

I'd say the most straightforward way is to modify your loss function to weight errors on the "bad" classes stronger in relation to other errors. Of course this is a trade-off at the cost of lost performance somewhere else. You generally cannot just get better performance on these classes for nothing.

For underrepresented classes I'd have recommended to oversample them, but according to your description that does not seem to be the issue.

$\endgroup$
7
  • $\begingroup$ I haven't tried modified loss functions yet; any tip/resource on how to do that in R? I'm using caret and I did try over- and under-sampling, and SMOTE, but they all resulted in worse performance. Also, does my additional model idea seem valid? $\endgroup$ – adatum Jan 25 '17 at 19:11
  • $\begingroup$ No idea about R, using python myself, sorry. I would just hand-code a suitable loss function. I'm not really a fan of adding additional models for it as it is somewhat indirect/messy, but I don't see why it shouldn't work in general. $\endgroup$ – Martin Krämer Jan 25 '17 at 19:14
  • 1
    $\begingroup$ Just want to add that if you do tweak the weights of the loss function to optimize performance on the data, this is a form of model selection/hyperparameter tuning. So, your final statistics/evaluation would have to take that into account to avoid being overoptimistically biased by 'peeking'. $\endgroup$ – user20160 Jan 25 '17 at 20:33
  • 1
    $\begingroup$ I don't mean to say that this is anything unusual. Seems analogous to the case where you're, say, tuning a regularization/penalty term (i.e. modifying the loss function) to maximize performance. You'd always do things like use an independent validation set for tuning, another independent test set to estimate final performance, etc. Hopefully the data are representative of the 'true' distribution. But, because there are a finite number of samples, it's still possible to overfit, produce overoptimistic estimates, etc. without these precautions (and, in some cases, even with them). $\endgroup$ – user20160 Jan 26 '17 at 19:26
  • 1
    $\begingroup$ btw I agree that your suggestion about class weights makes sense to try (+1) $\endgroup$ – user20160 Jan 26 '17 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.