0
$\begingroup$

I trained a Random Forest classification model to predict bioactivity for different protein targets. Both my training and test sets were highly imbalanced with ~99% of the majority class. Now that I'm trying to evaluate my predictions on the test set I observed that the sensibilities (SE) and specificities (SP) were in the range 0.80-0.98. I got very good Matthew's Correlation Coefficient (MCC) values (0.80-0.90) for some test sets, but others showed a considerabily lower MCC even with good SE and SP. Here's an example

Target 1:

SE = 0.911

SP = 0.998

MCC = 0.948

Target 2:

SE = 0.880

SP = 0.912

MCC = 0.321

I plotted the MCC x False positives and observed a logarithmic relationship (see the attatched plot). My hypothesis is that despite the low Type I error (high specificity) the high class imbalanced is penalizing too much my MCC; but the models still have a moderate accuracy. Is this correct or should I look at things in different way?

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.