2
$\begingroup$

I have ordered categories, and a prediction for the size of the categories. If the categories were unordered, I would use Cramer's V as a measure of the effect size. [Feel free to give an opinion on whether that's the best.] But I have ordered categories. What's the best effect size estimator in this case?

Here's an example with 9 categories:

enter image description here

With over 700 thousand observations in this case (often more) a significance test is beside the point. I could use Cramer's V, but this ignores the fact that these are ordered categories. I want to compare the fit of this data to the expectation with the fit of other data to the same expectation. What would you recommend?

As an aside, I note this related question is unanswered: Best effect size estimator in simple linear regression?

$\endgroup$
4
  • $\begingroup$ Are you interested in particular kinds of deviation from the expected? Or is any kind of deviation whatever equally important? $\endgroup$
    – Glen_b
    Nov 28, 2015 at 5:36
  • $\begingroup$ @Glen_b any deviation equally important. $\endgroup$
    – zbicyclist
    Nov 28, 2015 at 17:59
  • $\begingroup$ So the effect size you seek is one-sample, & a goodness of fit, is that correct? $\endgroup$ Nov 28, 2015 at 23:41
  • $\begingroup$ @gung That's correct. $\endgroup$
    – zbicyclist
    Nov 29, 2015 at 3:48

1 Answer 1

3
$\begingroup$

If you were more interested (for example) in deviations from expected that corresponded to a movement of probabilities up to higher categories or say to a polarization toward the extreme (relative to say an increase in the proportion of odd-numbered categories over even numbered categories which doesn't lead to much effect on say the median or the "spread"), then you can take advantage of the ordering to gain more sensitivity/attention to those kinds of deviation which relate to the order.

But since any deviation at all is equally of interest, then you're not particularly interested in deviations that relate to the ordering, so you can take no meaningful advantage of the fact that the categories are ordered.

Consequently for that sort of omnibus alternative, you would simply ignore the ordering and use measures of deviation that detect deviations from expectation in unordered categories.

Edit: I should mention the smooth alternatives for ordered categories that lead to generalizations of "smooth tests", which generalize from Neyman-Barton smooth tests in goodness-of-fit, but now expanded to include ANOVA-like problems for example. This includes work done by many authors but Rayner and Best are two of the major names in this line of work. As an example, for a chi-squared type goodness of fit test, you'd partition the statistic into orthogonal components -- linear, quadratic, cubic etc. If you are interested in detecting deviations from expected that relate to the ordering, the first few orthogonal components will capture that well (4 is a common choice, but if there's a lot of categories, 6 is also often used), and give good ability to see things like "shift in location" or "polarization to the extremes". However, if all kinds of deviations are of interest, you'd include all the components at higher polynomial orders (all the ones available for the number of categories you have), and you're back with the original order-ignoring chi-square statistic.

In a similar vein, the reason the Kolmogorov-Smirnov test has more power against alternatives of common interest than the chi-square in goodness of fit testing* is mostly because it takes some account of ordering in the data; but that involves having very low power against the high-order alternatives.

* (with categorical data, this requires appropriately dealing with the discreteness of the statistic, not using the tables designed for the continuous case)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.