0
$\begingroup$

I understand different statistical tools have their own pros and cons. I'm trying to find the most appropriate one for my situation.

I have a large, unbalanced data set and want to implement the chi-square test on it to test the independence of two categories (or more).

Since the size is large, only looking at p-value always give me a significant result which is not what I want. To use effect size like Cramer's V, the unbalance of the data set automatically gives me a low score.

The unbalance comes from the nature of the data, like number of cancer patients, so I don't think use sampling strategy will be a good idea, cause it changes the underlying distribution.

I'm wondering if there's any appropriate strength test or other method fitting into my situation? (A standard method will be preferred)

Any idea is appreciated:)

E.g., a way I'm trying is that instead of normalized by the number of total data like in Cramer's V, I use the number of minority part to normalize, so that it can be robust to the size of sample and deal with the imbalance issue. Of course its cons is the sensitivity w.r.t. the size of the minority part, which makes it more like a ratio test.

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ It's not clear what you mean by "strength test" and your use of "exact test" in relation to Cramer's V appears to be different to the conventional meaning of "exact" in relation to statistical tests, so I am unsure what you mean there as well. Are you able to explain what you are trying to achieve? (preferably without using a word like "test" at all) $\endgroup$
    – Glen_b
    Aug 28, 2018 at 0:23
  • $\begingroup$ Sorry, I do mean effect size instead of exact test. I've revised my question. For strength test, I mean any method which can give me a fair result indicating the real correlation level of the variables for my situation. $\endgroup$
    – G. Yu
    Aug 28, 2018 at 13:11
  • $\begingroup$ Now I really don't know what you mean by "fair result"; and the "real correlation level" is ambiguous. You can measure the strength of association in a contingency table by a number of different measures. $\endgroup$
    – Glen_b
    Aug 28, 2018 at 14:29
  • $\begingroup$ Yeah that's what I mean. I'm wondering among many different measures, which is the best or appropriate for my situation, cause it seems p-value and Cramer's V are not. $\endgroup$
    – G. Yu
    Aug 28, 2018 at 14:38
  • $\begingroup$ Yes, it appears you are trying to force Cramer's V to behave differently than it does. When you say your data is unbalanced, I assume you mean that in the contingency table you are treating either the columns or rows as groups, and you want to look at the proportions of counts among those groups, and you want the proportions to be the focus, so that the same proportions with different counts are treated as the same value. In a 2 x 2 table, Cohen's h acts this way, or odds ratio could be used. But it's not clear to me how you would create a similar measurement for larger tables. $\endgroup$
    – Sal Mangiafico
    Aug 29, 2018 at 1:37

1 Answer 1

Reset to default
0
$\begingroup$

I'm wondering if there's any appropriate strength test or other method fitting into my situation?

Maybe you should not look for some intricate statistical test. Just specify the sample size, emphasizing that it is large and therefore the p-value is knowingly low.

After that, report the p-value together with the effect size. This is a good practice. The reader gets the opportunity to draw her own conclusions about the significance of your results.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Cramer's V is an appropriate effect size statistic for a chi-square test of association. $\endgroup$
    – Sal Mangiafico
    Aug 28, 2018 at 13:13
  • $\begingroup$ Thanks for your advise! But I do want to compare different results for different variables and pick certain threshold to pick the most powerful one, like in CHAID algorithm. $\endgroup$
    – G. Yu
    Aug 28, 2018 at 13:13
  • $\begingroup$ Yeah I know Cramer's V is a classical choice. But for my case, the data is unbalanced, which results in an automatic low Cramer's V score. E.g. the number of data is ~ 100,000, but the reduced chi-square score is at most a few hundred, much less than the classical threshold like 0.1, 0.2, 0.3. $\endgroup$
    – G. Yu
    Aug 28, 2018 at 13:21
  • $\begingroup$ Thanks for the advise! Reporting p-value and effect size is definitely a good choice. But right now I'm the reader, I want to use certain quantities to help me dig the information from the data using computers. And in my case, both p-value and effect size (Cramer's V) are too small to make any sense. $\endgroup$
    – G. Yu
    Aug 29, 2018 at 13:38
  • $\begingroup$ @G.Yu Why does a small effect size not make sense? $\endgroup$
    – Dave
    Jan 23, 2022 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.