Correct P values from Chi-squared test by sample size

Question

I am working with results from a chi-squared test of homogeneity, but I'm struggling with the idea behind this test, since the chi-squared statistic can only increase with the number of observations, and then some P values are returned that are more significant that others probably just because of a higher number of observations. Lets put an example:

Nobs <- c(34, 34, 41, 28)
Nexp <- c(26, 44, 45, 22)

The chi-square statistic here is 6.73 (P = 0.08, df = 3), calculated as:

((Nobs - Nexp)^2 / Nexp) %>% sum()

However, if I consider now a second set of data less biased from the null expectation (e.g. with Nobs/Nexp -1 closer to zero overall), with the same number of categories, but with much higher number of observations for each one of these, the value of chi-squared increases, just because I am relying on more data, eg:

Nobs2 <- c(180, 422, 341, 211)
Nexp2 <- c(212, 372, 348, 222)

Which gives chi-squared = 12.24 (P = 6.6e-3).

Is there a way to correct these P values by the underlying number of observations, so these are comparable?

Thank you

Answer 1

The effect you are observing --- that the p value decreases with a larger sample size for the same effect size --- is a feature of p values. That's just the way p values work.

In contrast, a proper effect size statistic will not increase as the sample size increases for the same effect size. It's a good practice to report effect size statistics, or some other measure of effect size (like proportions or differences in means).

For a chi-square test of association, common effect size statistics are phi and Cramer's V

For a chi-square goodness-of-fit test, there's no standard effect size statistic that I know of, except maybe just comparing observed proportions to theoretical proportions.

But also for a goodness-of-fit test we can make a modification of Cramer V statistic, taking the square root of the chi-square value divided by the sample size divided by the number of categories minus 1. See details at the documentation for this function in the rcompanion package, with the caveat that I am the author of this package, and the code below.

To demonstrate how effect sizes are independent of sample size, I'm going to create a third set of values which has the same proportions as Nobs and Nexp, but with three times the sample size.

You'll note the proportions and modified Cramer's V statistic are the same for Nobs and for Nobs3.

Nobs = c(34, 34, 41, 28)
Nexp = c(26, 44, 45, 22)
Pexp = Nexp / sum(Nexp)

chisq.test(Nobs, p = Pexp)

   ### Chi-squared test for given probabilities
   ###
   ### X-squared = 6.7262, df = 3, p-value = 0.08116

### Report proportions observed

Pnobs = Nobs / sum(Nobs)

Pnobs

   ### 0.2481752 0.2481752 0.2992701 0.2043796

Pexp

   ### 0.1897810 0.3211679 0.3284672 0.1605839

### Modified Cramer's V

Chi.sq    = chisq.test(x=Nobs, p=Pexp)$statistic
K         = length(Nobs)
N         = sum(Nobs)
CV        = sqrt(Chi.sq/N/(K-1))
names(CV) = "Modified Cramer's V"
CV

   ### Modified Cramer's V 
   ###           0.1279274  

# # # # # # # # # # # # # # # # # 

Nobs3 = Nobs * 3
Nexp3 = Nexp * 3
Pexp3 = Nexp3 / sum(Nexp3)

chisq.test(Nobs3, p = Pexp3)

   ### Chi-squared test for given probabilities
   ###
   ### X-squared = 20.179, df = 3, p-value = 0.0001559

### Report proportions observed

Pnobs3 = Nobs3 / sum(Nobs3)

Pnobs3

   ### 0.2481752 0.2481752 0.2992701 0.2043796

Pexp3

   ### 0.1897810 0.3211679 0.3284672 0.1605839

### Modified Cramer's V

Chi.sq    = chisq.test(x=Nobs3, p=Pexp3)$statistic
K         = length(Nobs3)
N         = sum(Nobs3)
CV        = sqrt(Chi.sq/N/(K-1))
names(CV) = "Modified Cramer's V"
CV

   ### Modified Cramer's V 
   ###           0.1279274