5
$\begingroup$

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in a classifier with less accuracy - the usual tradeoff. I am running the comparison on some 50 datasets, and I want to report the difference on the accuracy by using the heuristic.

1) I was first planning on using Cohen's h on the difference of the accuracy. $$ h = 2 \arcsin(ac1) - 2 \arcsin(ac2) $$

EDIT: I found two conflicting formulas for the Cohen h. The formula above is in book Statistical Strategies for Small Sample Research edited by Rick H. Hoyle

The second formula is $$ h = 2 \arcsin(\sqrt{ac1}) - 2 \arcsin(\sqrt{ac2}) $$ and the source seems to be Cohen himself at this link http://people.ucalgary.ca/~ramsay/cohen-effect-size-h-arcsin-transformation.htm.

END EDIT

(the intuition on the Cohen h is that it is much harder to improve the accuracy of a classifier from 0.90 to 0.95 than to improve another classifier from 0.70 to 0.75, that is, the effect size of the first change is larger than that of the second. Cohen's h theory actually talks about variance - the variance in the 0.90 is smaller than the variance around 0.7.)

2) But it hit me that to be even more fair, I must take into consideration how hard it is to have the accuracy of 0.9 or 0.70 to begin with. This is where the Kappa enters. Kappa measures how much above a "guessing" classifier is my accuracy (ac) $$ \kappa = {ac - guess\over{1-guess}} $$ where guess is the accuracy of guessing classifier that is aware of the proportion of the classes in the data $$ guess = pos*pos + neg*neg $$ where pos and neg are the proportion of positive and negative examples.

3) Here my first question. Is there an effect size measure for Kappa? Should I make the arcsin transformation or just use the average difference in kappa?

4) I also realized that I can use the variance of the accuracy or kappa for each classifier. I was originally planing in using a hold-out cross-validation, but if there is a use for the variance, I can use any other cross-validation that gives me an estimate of the variance (repeated hold-out, k-fold, bootstrap).

In this case I could use a Cohen d to report the difference in accuracy (or Kappa). Does this make sense?

Any thoughts?

Thanks

$\endgroup$
2
  • $\begingroup$ Did you read en.wikipedia.org/wiki/Cohen's_kappa, paragraph "Kappa maximum"? I expect that the ratio of observed kappa to its possible maximum in your table may be a measure of effect size. $\endgroup$
    – ttnphns
    Jul 25, 2014 at 9:58
  • $\begingroup$ Yes, it is a possibility, but I still dont know if a change of 0.05 in kappa is the same if the initial Kappa (or ratio to the max kappa) is 0.7 or 0.9. That is, the variance of Kappa (or Kappa ratio) is uniform across the range of values? This lack of uniformity of the variance (of the accuracy) that is behind Cohen's g arcsin transformation. $\endgroup$ Jul 28, 2014 at 23:06

1 Answer 1

1
$\begingroup$

Cohen's $\kappa$ describes the reduction in error rate of your classifier compared to that of random label assignment. That is, random label assignment leads to a certain amount of garbage (misclassifications), and Cohen's $\kappa$ describes how much of that garbage is removed. Cohen's $\kappa$, thus, has a similar interpretation to what I discuss here and here about comparing to a baseline error rate (though Cohen's $\kappa$ seems to calculate that baseline eror rate a bit differently than I would in that setting).

By having the Cohen's $\kappa$ of the two models, you can compare how much garbage each is able to remove, and you can decide, based on their difference or ratio, what is acceptable. If the slow model removes $80\%$ of the garbage and the fast model removes $79\%$ of the garbage (so almost $99\%$ of the garbage the slow model removes is removed by the fast model), the fast model that is almost as good might be acceptable. If the fast model only removes $50\%$ of the garbage (so less than $63\%$ of the garbage that the slow model removes), the speed of the slow model might be acceptable given how much better it works.

(I get the $99\%$ and $63\%$ from $79/80$ and $50/80$, respectively.)

Therefore, I would not bother with the arcsine transformation. Cohen's $\kappa$ already has a straightforward interpretation and ability to compare models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.