1
$\begingroup$

I'm running a Chi Sq test with 2 binary variables. The data are large (>1 million rows) and not balanced (rare event). The test is statistically significant (p < .0001) and the Cramer's V is very small (.006). I took this to mean there is no relationship and the p value is due to such a large sample size/power. However, the odds ratio is 4.4.

I'm trying to understand how one effect size (Cramer's V) can be so unlike another (Odds Ratio)? Is the Cramer's V sensitive to data imbalance? enter image description here

$\endgroup$

2 Answers 2

2
$\begingroup$

This is a good question with a few moving pieces, so my answer may be a bit circuitous. But given your setup, I think it's important to hit all the relevant stops along the way.

There are boundaries that we have to first establish as to what these tests can and cannot tell us. First off, class imbalance alone has no impact your chi-square test results. Try to artificially create imbalance in your "flag class" ranging from 1:2 to 1:35 to 1:1000, etc. and you could hypothetically create 2x2 matrices with chi-square statistics close to zero (likely unassociated). This in mind, I'm looking at your data, and not actually too worried about the class imbalance per se. That won't "drive" anything here.

What will matter is your actual overall sample size (your n) and the number of columns/rows you have. Jumping ahead, this will determine whether your should be relying more on the OR or Cramer's V.

Chi-square statistics tell you nothing about the effect size (strength of association), only of the association occurring by chance. Big differences between classes (i.e. white and black) in matrices with low overall sample sizes can produce non-significant chi-square results. And in large sample sizes, even the smallest differences between groups can generate significant chi-square results, with p-values well below 0.0001. This is because the p-value indicates how likely you will obtain the chi-square statistic by chance. Higher sample sizes makes it extremely unlikely, and so differences between groups becomes more sensitive.

Whereas Chi-Square provides indication of an association, Cramer's V and Odds Ratio provide an estimate of the strength of the association. However, which to use depends on a few things:

  1. Generally, OR is best used on 2x2 contingency tables. Cramer's V can conveniently provide effect size estimates on tables of multiple dimensions with a "correction" for large sample sizes. But this has the effect of producing low correlation measures, even for highly significant results.

  2. Cramer's V also assumes that both columns (i.e. black/white and flag/no flag) can be thought of as independent variables. It is a symmetrical test, meaning V(x,y)=V(y,x). I don't believe this to be true in your setup, as you're trying to describe one as having a dependent effect on the other, or the odds of some kind of success in the flag group vs. odds of success in the non-flag group.

  3. Cramer's V is also sensitive to large sample sizes, which you have, regardless of the class imbalance that is also present in your dataset. Contrary to the chi-square statistic, larger group differences in smaller samples will show larger associations (as they should), and smaller group differences within larger samples will show smaller associations (as they should). This is why your Cramer's is low (0.0064) while your Chi-Square is high (60.08, p-value <0.001).

Therefore, I would tend to summarize your results using the OR and Chi-Square statistics, and put less emphasis on the Cramer's V for the reasons mentioned above. Your 95% CI limits also look good for the OR.

However, I would also stress that OR assumes the underlying distribution of your data is normal, and that this sample of observations you've selected is also random (Berkson's Fallacy). You can't make inferences on entire populations using restricted observation samples.

$\endgroup$
3
$\begingroup$

I happened to come across this interesting question, and wanted to provide my 2 cents, expanding on the answer already given. I will assume that the provided data meet the requirement for the chi2 test.

What I would like to point out is that Cramer's V, as well as other coefficients such as Phi, C, and even Goodman-Kruskal's lambda, is sensitive to both the magnitude of the association AND to the marginal distribution.

In the dataset provided by the user, the marginals are pretty unbalanced (skewed). It is not by chance, in fact, that V proves pretty small (0.006383) while the OR is indeed not "tiny" (4.44).

The imbalance in the marginal sums, which does not affect the OR, would indeed account for the small value of V. Some scholarly sources suggest to control for the imbalance by computing the coefficient on the standardised table. As matter of fact, V (computed on the standardised version of the user's table) turns out to be 0.356. For a 2x2 table, it could be interpreted as an association of medium strength (according to the Cohen's thresholds), and is more "in line" with the magnitude of the association as measured by the OR.

What do I mean by "standardised table"? A standardised table is a table where we re-express the counts so to achieve predetermined sets of marginal sums. This is done via Iterative Proportional Fitting (IPF). For instance, the row and column marginals can be equal to fractions out of 100. Or, one can retain the table's grand total and set the row (and column) marginals to the grand total divided by the number of rows (and columns).

Using this last approach, the standardised version of the table provided by the user would be:

enter image description here

Note that we are "re-expressing" the counts, keeping the association structure is unaltered (OR 4.44). The marginals are not unbalanced.

Essentially, the way I see the issue is: given the observed V, is it what it is (e.g., "small", like in the example provided by the user) because the association is indeed "tiny" or also because the imbalance in the marginal distribution is "getting in the way"? For one thing, the discrepancy between V and the OR should ring as a warning bell. One might wonder whether in such contexts (i.e., in a table with such a marginal imbalance) measures that are sensitive to imbalances in the marginal sums should not be used, and more "resistant" measures should be preferred instead. Or, on the other hand, we could still stick to V "but" controlling for the marginal skewed distribution.

The issue becomes even more cogent when comparing Vs across tables, for example.

There is a nice section on the effect of marginal distribution on nominal association coefficients, and on table's standardisation, in the following book: Reynolds, H. T. (1977). The Analysis of Cross-Classifications. The Free Press. ISBN 0029263905.

Also, the seminal Agresti's book on Categorical Data Analysis features a paragraph on tables standardisation via IPF.

$\endgroup$
8
  • $\begingroup$ What do you call the standardized table? $\endgroup$
    – J-J-J
    Feb 28 at 11:41
  • 1
    $\begingroup$ I have edited my answer in order to address your comment. $\endgroup$
    – NewAtGis
    Feb 28 at 12:44
  • $\begingroup$ Thanks, I see. +1 for the explanation and the reference. However, as you largely altered the data and the resulting Cramér's V, I find it dubious that you can compare it to Cohen's rules of thumb, which are based on non-altered calculations (but maybe there's something justifying it in the reference you mention?). Anyway, Cohen's rules of thumb are themselves are subject to criticism, and ultimately may not be useful at all to judge the relevance (or the "strength") of an effect size relative to the context of a given study. $\endgroup$
    – J-J-J
    Feb 28 at 13:19
  • $\begingroup$ (cont'd) In addition, in this specific case of a 2x2 table, I find that your solution introduces an unnecessary complication to try to correct a known issue with Cramér's V (or phi) in 2x2 tables (see stats.stackexchange.com/a/632240/164936). When phi is inadequate for the question we want to answer, the odds ratio (or the risk ratio) is a simpler and more intuitive solution. $\endgroup$
    – J-J-J
    Feb 28 at 13:27
  • $\begingroup$ (cont'd) Also, I wonder about the properties of the altered Cramér's V you suggest: what kind of inference can we make from it? Can we compute reliable confidence intervals around it? etc. $\endgroup$
    – J-J-J
    Feb 28 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.