2
$\begingroup$

In a numerical study I want to assess the performance of a binary classification algorithm learned repeatedly using different training datasets and applied to varying test datasets. The distribution of the binary class variable in the test datasets is often very skew in this analysis, frequently featuring only instances from one class, i.e. {pos,pos,pos,...} or {neg,neg,neg,...}.

The misclassification error rate is not meaningful here, because it depends on the distribution of the class variable. Therefore I wanted to use Matthews correlation coefficient (MCC). However, the latter is not calculable in specific situations, in which the nominator and denominator in the formula are both zero. More precisely this happens in the following situations: 1.) both real values and predictions are ALL positive/negative; 2.) real values are all negative and predictions all positive; 3.) real values are all positive and predictions all negative; 4.) predictions are all negative; 5.) real values are all negative/positive.

In situation 1.) one could just set MCC to 1, because no errors are made and in situations 2.) and 3.) one could set it to -1, because the predictions totally contradict the true values. However, in situations 4.) and 5.) I do not know how to proceed. Slightly off-topic remark: When, analoguous to situation 4.), the predictions are all negative, a zero value for MCC results, which seems not meaningful in most situations. For example, consider that there are 100 negatives and 1 positive - when the classificator assigns negatives to all cases it seems not to perform bad, but receives a MCC-value of zero.

My questions are: Is there a version of Matthews correlation coefficient which can deal with the situation where there are only cases from one class? If not, would there be an alternative, which - like MCC - does not depend on the distribution of the class variable?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.