There are many related questions here and elsewhere, but I found no satisfying answer.

There are plenty of related tests and concepts: Z-test, chi-squared test, G-test, McNemar's test, Fisher's exact, and then the Odds Ratio and Relative Risk, and potential arcsine tranformations...

I do have vague ideas about how some of these apply only for specific contexts, e.g. Fisher's exact is for small sample, McNemar's test is for paired sample, and if I understand well Odds Ratio and Relative Risk are rather descriptive measures than tests - but how all these relate to each other is unclear.

Most importantly, I just don't know which test to use to compare two simple ratios. Let's say I have two independent samples, n1 = 100, n2 = 120, with a ratio of whatever in either: ratio1 = 0.75, ratio2 = 0.80.

What test should I use and why? I'd also want related effect size and confidence interval.

My specific interest is actually to compare two classification accuracies: e.g., 75 out of 100 people classified correctly (so sum of all true pos. and true neg. cases relative to all cases) with one method, and 80 out of 120 with another method. I know I could compare AUCs but that's not what I want to do here: I want to compare the best accuracies, obtained at the optimal threshold.

My best guess so far is to use a Z-test with Cohen's h - because that's what I understand best (in particular Cohen's h, based on Cohen, 1988). For the above example, the result would be: Z = -0.89, p = 0.376 h = -0.120, 95% CI [-0.385, 0.145]. How correct is this? I'm quite sure about Cohen's h, but I don't know about the Z-test.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.