0
$\begingroup$

I wondered about the relationship between a confidence interval for Cramer's V (I used the rcompanion package and the command cramerV(table, ci = TRUE)) and the result of the chi-squared test (I used chisq.test(data$x, data$z, correct = FALSE)) with the same data. My initial thought was, that IF the 95% CI of Cramer's V includes zero, we do not reject the null hypothesis of there being no association. I expected the Chi-square test, to tell me the same thing as this CI. But that is not true.

Here is a reproducible example using R:

library("tidyverse")
library("rcompanion")

set.seed((3111965))
n <- 60

data <- c(1:n) %>% as_tibble() %>% rename(., RNR = value)
data$x <- sample(c(1,2,3), n, replace = TRUE)
data$x <- as.factor(data$x)
data$z <- ifelse(data$x == 1, 3, ifelse(data$x==3, 2, 1))
data$temp <- sample(c(1,2,3), n, replace = TRUE)
data$binom <- rbinom(n,1,0.80) # change 0.80 if you want x and x to be very/not different
data$z <- ifelse(data$binom == 1, data$temp, data$z)

table <- table(data$x, data$z)
cramerV(table, ci = TRUE)

chisq.test(data$x, data$z, correct= FALSE) # I wanted to avoid the correction

This gives a CI Cramer's V not including zero, and a Chi square telling me the chance of finding this Chi square while there is actually NO association in the data is bigger than 5% (not significant).

I am probably overlooking something very basic, but I do not understand.

EDIT

After playing/reading a bit more, it appears that Cramér's V can be a heavily biased estimator of its population counterpart and will tend to overestimate the strength of association. By adding bias.correct = TRUE in the estimation in R (see Wikipedia for the formula underlying the bias correction), it more often 'includes zero' (which is of course the lowest value possible), and it seems the outcome of the chi-square test and the CI's for Cramers V are aligned. So I think, I solved the issue myself. Am I correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

The Cramer's V statistic is always non-negative. Because of this, the bootstrapped confidence interval by the percentile method will never cross zero. So it's probably not a reasonable method to use for hypothesis inference.

The percentile method is the default for rcompanion::cramerV().

I don't know if there is another method to calculate the confidence interval for Cramer's V that may be applicable for inference.

You might compare this behavior to an effect size statistic like phi or r, that can cross zero.

From the function documentation:

Because V is always positive [or zero], if type="perc", the confidence interval will never cross zero. In this case, the confidence interval range should not be used for statistical inference.

www.rdocumentation.org/packages/rcompanion/versions/2.4.18/topics/cramerV

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ The documentation also says: "However, if type="norm", the confidence interval may cross zero." This suggests that that version CAN be used in the context of statistical inference. I am not into bootstrapping, so I do not understand the difference between perc and norm when calculating a confidence interval. But in sum: the combination of the a) bias correction and b) using the 'norm' way of calculating the CI are necessary and sufficient for completely aligning the outcomes of the CI for Cramers V and the Chi square test? $\endgroup$
    – Henk van der Kolk
    Commented Aug 18, 2022 at 13:40
  • $\begingroup$ I don't know. Personally, I wouldn't rely on confidence intervals for any effect size statistic that can't be negative for hypothesis testing or inference. $\endgroup$
    – Sal Mangiafico
    Commented Aug 18, 2022 at 14:01
  • $\begingroup$ Still, it can be zero, so that should be enough. I just want to make sure that CI and Chi-square are giving me the same result (otherwise I am starting to doubt my basic understanding of inferential statistics). $\endgroup$
    – Henk van der Kolk
    Commented Aug 18, 2022 at 14:30
  • $\begingroup$ @HenkvanderKolk , you might want to run some simulations on randomly selected data to compare the p-value from the hypothesis test and the range in the confidence intervals. Of course, there are options in the chi-square test (like continuity correction). Bootstrapped confidence intervals will vary inherently because they are generated from randomly selected samples, and there are different options for constructing the bootstrapped confidence intervals ("norm", "perc", "bca", and so on). And, as you note, you can or not apply the bias correction to Cramer's V. (con't) $\endgroup$
    – Sal Mangiafico
    Commented Aug 18, 2022 at 15:04
  • 1
    $\begingroup$ Thanks, I now see much better the relationships between the various topics addressed. However, "the CI for Cramer's V isn't too far off the conclusion from chisq.test()." assures me that everything is still fine. I will run some simulations later, see what happens. All these 'corrections' (both for the chi square test and for Cramers V) make the whole idea less transparent. I was hoping it was simpler than this. But I guess it never is. $\endgroup$
    – Henk van der Kolk
    Commented Aug 18, 2022 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.