I trained a Random Forest classification model to predict bioactivity for different protein targets. Both my training and test sets were highly imbalanced with ~99% of the majority class. Now that I'm trying to evaluate my predictions on the test set I observed that the sensibilities (SE) and specificities (SP) were in the range 0.80-0.98. I got very good Matthew's Correlation Coefficient (MCC) values (0.80-0.90) for some test sets, but others showed a considerabily lower MCC even with good SE and SP. Here's an example

Target 1:

SE = 0.911

SP = 0.998

MCC = 0.948

Target 2:

SE = 0.880

SP = 0.912

MCC = 0.321

I plotted the MCC x False positives and observed a logarithmic relationship (see the attatched plot). My hypothesis is that despite the low Type I error (high specificity) the high class imbalanced is penalizing too much my MCC; but the models still have a moderate accuracy. Is this correct or should I look at things in different way?

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.