I was wondering something about imbalanced datasets. It's not the usual 'what should I do with my imbalanced training set'.
I have an imbalanced training set with
1:2 = positive:negative examples. I have a test set with about 1:13 ratio as well (I need a separate test set since I would like to compare my model to existing models. The test set was compiled from incoming examples after the compilation of the training set, hence the different ratio.)
Now, I have trained a model and I have created a confusion matrix (CM) from the CV and from the test set as well.
The CM of the test set looks like this:
observed true observed false pred. true 94 90 pred. false 29 1554
It has a sensitivity = 0.76, specificity = 0.95, MCC = 0.59, precision = 0.51, F1 = 0.61 etc.
However, after that I realised that these measures very much depends on the ratio of the two classes. I have tried two scenarios.
First, I have balanced the test set like thr training set, having a 1:2 ratio. I got this:
observed true observed false pred. true 94 13 pred. false 29 233
It has a sensitivity = 0.76, specificity = 0.95, MCC = 0.74, precision = 0.88, F1 = 0.81 etc. It is clear, that the first two did not changed, however others greatly improved.
Then, I trid a 1:1 balanced dataset:
observed true observed false pred. true 94 7 pred. false 29 116
It has a sensitivity = 0.76, specificity = 0.95, MCC = 0.72, precision = 0.93, F1 = 0.84 etc.
To be clear, I totally understand why sensitivity and specificity remains the same, and not affected by sample ratio. However, MCC is widely used to compare models and to evaluate model's performance. I know I would like to be fair in model building and comparisons and would not like to use the scenario that best fits my goals if that is not the proper one. So which one (or any other) is reasonable to use?
Thanks for answers.