I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in a classifier with less accuracy - the usual tradeoff. I am running the comparison on some 50 datasets, and I want to report the difference on the accuracy by using the heuristic.

1) I was first planning on using Cohen's h on the difference of the accuracy. $$ h = 2 \arcsin(ac1) - 2 \arcsin(ac2) $$

EDIT: I found two conflicting formulas for the Cohen h. The formula above is in book Statistical Strategies for Small Sample Research edited by Rick H. Hoyle

The second formula is $$ h = 2 \arcsin(\sqrt{ac1}) - 2 \arcsin(\sqrt{ac2}) $$ and the source seems to be Cohen himself at this link http://people.ucalgary.ca/~ramsay/cohen-effect-size-h-arcsin-transformation.htm.


(the intuition on the Cohen h is that it is much harder to improve the accuracy of a classifier from 0.90 to 0.95 than to improve another classifier from 0.70 to 0.75, that is, the effect size of the first change is larger than that of the second. Cohen's h theory actually talks about variance - the variance in the 0.90 is smaller than the variance around 0.7.)

2) But it hit me that to be even more fair, I must take into consideration how hard it is to have the accuracy of 0.9 or 0.70 to begin with. This is where the Kappa enters. Kappa measures how much above a "guessing" classifier is my accuracy (ac) $$ \kappa = {ac - guess\over{1-guess}} $$ where guess is the accuracy of guessing classifier that is aware of the proportion of the classes in the data $$ guess = pos*pos + neg*neg $$ where pos and neg are the proportion of positive and negative examples.

3) Here my first question. Is there an effect size measure for Kappa? Should I make the arcsin transformation or just use the average difference in kappa?

4) I also realized that I can use the variance of the accuracy or kappa for each classifier. I was originally planing in using a hold-out cross-validation, but if there is a use for the variance, I can use any other cross-validation that gives me an estimate of the variance (repeated hold-out, k-fold, bootstrap).

In this case I could use a Cohen d to report the difference in accuracy (or Kappa). Does this make sense?

Any thoughts?


  • $\begingroup$ Did you read en.wikipedia.org/wiki/Cohen's_kappa, paragraph "Kappa maximum"? I expect that the ratio of observed kappa to its possible maximum in your table may be a measure of effect size. $\endgroup$ – ttnphns Jul 25 '14 at 9:58
  • $\begingroup$ Yes, it is a possibility, but I still dont know if a change of 0.05 in kappa is the same if the initial Kappa (or ratio to the max kappa) is 0.7 or 0.9. That is, the variance of Kappa (or Kappa ratio) is uniform across the range of values? This lack of uniformity of the variance (of the accuracy) that is behind Cohen's g arcsin transformation. $\endgroup$ – Jacques Wainer Jul 28 '14 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.