Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm running a Chi Sq test with 2 binary variables. The data are large (>1 million rows) and not balanced (rare event). The test is statistically significant (p < .0001) and the Cramer's V is very small (.006). I took this to mean there is no relationship and the p value is due to such a large sample size/power. However, the odds ratio is 4.4.
I'm trying to understand how one effect size (Cramer's V) can be so unlike another (Odds Ratio)? Is the Cramer's V sensitive to data imbalance? 
This is a good question with a few moving pieces, so my answer may be a bit circuitous. But given your setup, I think it's important to hit all the relevant stops along the way.
There are boundaries that we have to first establish as to what these tests can and cannot tell us. First off, class imbalance alone has no impact your chi-square test results. Try to artificially create imbalance in your "flag class" ranging from 1:2 to 1:35 to 1:1000, etc. and you could hypothetically create 2x2 matrices with chi-square statistics close to zero (likely unassociated). This in mind, I'm looking at your data, and not actually too worried about the class imbalance per say. That won't "drive" anything here.
What will matter is your actual overall sample size (your n) and the number of columns/rows you have. Jumping ahead, this will determine whether your should be relying more on the OR or Cramer's V.
Chi-square statistics tell you nothing about the effect size (strength of association), only of the association occurring by chance. Big differences between classes (i.e. white and black) in matrices with low overall sample sizes can produce non-significant chi-square results. And in large sample sizes, even the smallest differences between groups can generate significant chi-square results, with p-values well below 0.0001. This is because the p-value indicates how likely you will obtain the chi-square statistic by chance. Higher sample sizes makes it extremely unlikely, and so differences between groups becomes more sensitive.
Whereas Chi-Square provides indicates an association, Cramer's V and Odds Ratio provide an estimate of the strength of the association. However, which to use depends on a few things:
Generally, OR is best used on 2x2 contingency tables. Cramer's V can conveniently provide effect size estimates on tables of multiple dimensions with a "correction" for large sample sizes. But this has the effect of producing low correlation measures, even for highly significant results.
Cramer's V also assumes that both columns (i.e. black/white and flag/no flag) can be thought of as independent variables. It is a symmetrical test, meaning V(x,y)=V(y,x). I don't believe this to be true in your setup, as you're trying to describe one as having a dependent effect on the other, or the odds of some kind of success in the flag group vs. odds of success in the non-flag group.
Cramer's V is also sensitive to large sample sizes, which you have, regardless of the class imbalance that is also present in your dataset. Contrary to the chi-square statistic, larger group differences in smaller samples will show larger associations (as they should), and smaller group differences within larger samples will show smaller associations (as they should). This is why your Cramer's is low (0.0064) while your Chi-Square is high (60.08, p-value <0.001).
Therefore, I would tend to summarize your results using the OR and Chi-Square statistics, and put less emphasis on the Cramer's V for the reasons mentioned above. Your 95% CI limits also look good for the OR.
However, I would also stress that OR assumes the underlying distribution of your data is normal, and that this sample of observations you've selected is also random (Berkson's Fallacy). You can't make inferences on entire populations using restricted observation samples.
