2
$\begingroup$

I have a dataset with four numerical attributes and a class (target) variable. There is an enormous imbalance between positive and negative instances according to class variable. To cope with imbalance I use undersampling strategy; I randomly delete negative instances. Then I applied several machine learning algorithms and obtain quite promising results.

However, the reviewer suggests that I should apply baseline classifier (e.g., ZeroR) to assess the classifier results. But isn't such approach misleading? Accuracy of ZeroR will be 0.5 by default. What do you suggest?

$\endgroup$

1 Answer 1

1
$\begingroup$

The point of comparing to such a simple model that achieves accuracy according to the class ratio is the point of doing the comparison. You have heavy imbalance in your problem, so you can achieve high accuracy by predicting the majority category every time. You should check if you can do better than that. Comparing to this kind of baseline is analogous to $R^2$ in regression and makes a lot of intuitive sense. Your job is to make a classifier that is accurate (it probably isn't, but that's a separate issue). For your model to be worthwhile, you must be able to outperform some jerk who guesses the majority category every time.

You probably do not have to use any kind of artificial balancing in your pipeline, least of all downsampling that discards precious data. You can get a model that achieves $99.5\%$ accuracy and put that in context. If your baseline model predicts the majority category every time and can achieve accuracy of $99.9\%$, then you actually have an enormous increase in the error rate: you have increased the error rate by $400\%!$ Below, I show this in software that implements what I sometimes call $R^2_{\text{accuracy}}$, for reasons I explain here.

r2acc <- function(e1, e0){

  # e1: error rate of your model
  # e0: error rate of the baseline model

  return(
    1 - (e1/e0)
  )
}
r2acc(0.005, 0.001) # 0.005 corresponds to 99.5% accuracy, 0.001 to 99.9% accuracy

$$ R^2_{\text{accuracy}} = 1 - \dfrac{ \text{Error rate of the model under consideration} }{ \text{Error rate of a model that naïvely predicts the majority class every time} } $$

While there are still issues with using hard classifications instead of using the predicted probabilities, at least this $R^2_{\text{accuracy}}$ statistic picks up that a model with an impressive-looking $99.5\%$ accuracy is quite poor when the imbalance means that $99.9\%$ of the cases belong to one category so that a model could achieve $99.9\%$ accuracy by being a jerk and guessing that majority category every time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.