1
$\begingroup$

I was wondering something about imbalanced datasets. It's not the usual 'what should I do with my imbalanced training set'.

I have an imbalanced training set with 1:2 = positive:negative examples. I have a test set with about 1:13 ratio as well (I need a separate test set since I would like to compare my model to existing models. The test set was compiled from incoming examples after the compilation of the training set, hence the different ratio.)

Now, I have trained a model and I have created a confusion matrix (CM) from the CV and from the test set as well.

The CM of the test set looks like this:

               observed true       observed false
pred. true          94                  90
pred. false         29                 1554

It has a sensitivity = 0.76, specificity = 0.95, MCC = 0.59, precision = 0.51, F1 = 0.61 etc.

However, after that I realised that these measures very much depends on the ratio of the two classes. I have tried two scenarios.

First, I have balanced the test set like thr training set, having a 1:2 ratio. I got this:

               observed true       observed false
pred. true          94                  13
pred. false         29                  233

It has a sensitivity = 0.76, specificity = 0.95, MCC = 0.74, precision = 0.88, F1 = 0.81 etc. It is clear, that the first two did not changed, however others greatly improved.

Then, I trid a 1:1 balanced dataset:

               observed true       observed false
pred. true          94                   7
pred. false         29                  116

It has a sensitivity = 0.76, specificity = 0.95, MCC = 0.72, precision = 0.93, F1 = 0.84 etc.

To be clear, I totally understand why sensitivity and specificity remains the same, and not affected by sample ratio. However, MCC is widely used to compare models and to evaluate model's performance. I know I would like to be fair in model building and comparisons and would not like to use the scenario that best fits my goals if that is not the proper one. So which one (or any other) is reasonable to use?

Thanks for answers.

$\endgroup$
1
$\begingroup$

I think if in actual use your classifier is used on a set with a 1:13 imbalance, you should validate on a set with the same imbalance.

As for which measure to use, all these can be understood as simple cost functions, where each misclassfication is weighted equally heavy, or not at all, as precision doesn't penalize false negatives. The best classifier is then the one that minimizes cost.

In practice, these simple cost functions are often not realistic. Perhaps you can estimate a reasonable cost (in currency or in a relative way) of misclassifications, and use that to find the "best" model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.