Based on the following relationship between Matthew's Correlation Coefficient (MCC) and Chi Square:
(MCC is the Pearson product-moment Correlation Coefficient)
Is it possible to conclude that:
Imbalanced Binary Classification Problem, N = 1000, and χ² >= 3.85 (p < 0.05, df = 1)
Following MCC is significant:
MCC >= sqrt ( 3.85 / 1000 ) which is MCC >= 0.06
When comparing two algorithms (A, B) with trials of 100 times:
IF mean (MCC_A1..MCC_A100) - mean(MCC_B2..MCC_B100) > 0.06 THEN:
A significantly outperforms B
Thanks in advance!
Edit ROC curves provide an overly optimistic view of the performance for imbalanced binary classification
Regarding Threshold, I'm not a big fan of not using it, as finally one have to decide for a threshold, and quite frankly, that person has no more information than me to decide upon. Hence, providing PR or ROC curves are just for the sake of circumventing the problem for publishing.