1
$\begingroup$

I have a binary classification task with 10 ordinal features and 45,000 records, 8,500 of which should be classified as positive. To evaluate the classifier I build, I also build one on (a) the same classes but no features (i.e., random guessing); (b) the same records with all features but the classes shuffled (i.e., do we not see patterns in randomness? Could we use this classifier to classify some random distribution?). The classifier would be better if it scores better than the ones built with (a) and (b), of course. I use cross validation to measure the accuracy of the classifiers.

I have done test runs using (1) all data and (2) a randomly selected subset of the negative data to give 8,500 records of both classes. In run (2), my classifier scores significantly better than (a) and (b); 0.59 vs. 0.50 with a standard deviation of 0.04 vs. 0.00/0.01. In run (1), all three classifiers classify everything as negative, giving equal scores of 0.81 (standard deviation 0.38).

What does this mean? Is my classifier bogus? Or do the settings of the classifier (see below) not work for unbalanced data? Or is there not enough information to tell, in which case: what should I find out to be able to answer these questions?

The classifier is a decision tree with depth 4 and a minimum of 40 samples per leaf.

$\endgroup$
1
$\begingroup$

You should not use accuracy as your metric unless it is well above the level of imbalance. In your case this would be the 81% you get when down sampling. A typical alternative metric to use is the area under the ROC curve but it depends on your goal in general. This is a very common and well researched problem. Just google "imbalanced classification"

My favorite methods to solve this are the Balanced Bagging Classifier or Hellinger distance trees. However, you can get very good results with gradient boosted decision trees which is a much more available algorithm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.