3
$\begingroup$

I'm comparing performance of different classifiers on the data sets derived from financial markets, getting different accuracy and precision measures but Matthews correlation coefficient and Kappa statistic seldom exceeds 0.2. My data sets are quite big like 20000x170 so it should not be a problem with not enough of data.

So my question is: what value of MCC and Kappa can be considered as a 'good classifier' on such data.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I presume your are from Poland?

I do not think it is possible to answer for your question. Even in some statistics books about measures like AUC it is said it will be domain specific.

My classification models had recently their MCC in range of 0.2-0.25 with AUC bit over 0.7.

MCC can be calculated only if you specify a cutoff point when you have a probabilistic prediction.

But besides measures of accuracy for classification you might have a explicit cost/benefit measures for false positives and false negatives which might change cutoff point taken from the purely information point of view.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Yes, I'm from Poland. You said 'MCC can be calculated only if you specify a cutoff point when you have a probabilistic prediction.' What you mean by that and why probalistic prediction ?? According to Wikipedia is just a way to describe confusion matrix and as an input uses TN,TP,FN,FP $\endgroup$
    – Krzysztof Fajst
    Dec 2, 2013 at 12:34
  • $\begingroup$ There are two possibilities 1) you have a classification method which gives directly class labels 2) you have a classification method which gives a probability to belong to a class. Personally I prefer probabilistic prediction because you can have an estimate of variation and costs/benefits related to making a decision on different levels of probability to belong to a class. With only class labels, you cannot do that. $\endgroup$
    – Analyst
    Dec 3, 2013 at 6:39
  • $\begingroup$ But there might not be an answer for your question related to the level of these measures which you like to report. Some processes can easily be so difficult to predict with information only available publicly. Of course if you have inside information, then I think it could be possible to improve classification... :) $\endgroup$
    – Analyst
    Dec 3, 2013 at 6:44
  • $\begingroup$ thanks for info but i still don;t understand why you said 'MCC can be calculated only if you specify a cutoff point when you have a probabilistic prediction. ' My data sets are generated by high frequency FOREX rates (1min sampling) so no inside information. It seems that performance of classifiers is very unstable on this data i.e one day they perform great and another very bad $\endgroup$
    – Krzysztof Fajst
    Dec 3, 2013 at 11:17
  • $\begingroup$ Hi Analyst, please don't sign your posts with "Analyst". See this section of the help, under the heading "Do not use signature, taglines, or greetings." for an explanation of why. I have removed it for you in this case. Krzysztof, I have done the same with yours. $\endgroup$
    – Glen_b
    Dec 4, 2013 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.