Correct P values from Chi-squared test by sample size

I am working with results from a chi-squared test of homogeneity, but I'm struggling with the idea behind this test, since the chi-squared statistic can only increase with the number of observations, and then some P values are returned that are more significant that others probably just because of a higher number of observations. Lets put an example:

Nobs <- c(34, 34, 41, 28)
Nexp <- c(26, 44, 45, 22)

The chi-square statistic here is 6.73 (P = 0.08, df = 3), calculated as:

((Nobs - Nexp)^2 / Nexp) %>% sum()

However, if I consider now a second set of data less biased from the null expectation (e.g. with Nobs/Nexp -1 closer to zero overall), with the same number of categories, but with much higher number of observations for each one of these, the value of chi-squared increases, just because I am relying on more data, eg:

Nobs2 <- c(180, 422, 341, 211)
Nexp2 <- c(212, 372, 348, 222)

Which gives chi-squared = 12.24 (P = 6.6e-3).

Is there a way to correct these P values by the underlying number of observations, so these are comparable?

Thank you

• "The chi-squared statistic can only increase with the number of observations", this is the principle of statistical power: differences (even small) in large sample sizes are less likely due to sampling fluctuations than in small sample sizes, hence you get smaller p-values. Why would you want to correct the p-value? Feb 7 '20 at 16:12
• I feel that otherwise the comparison between the two P values is not fair. Or at least that differences in these P values are is not only a reflection of the differences between the two samples (their composition), but also of the differences in sample size for one and another. Ideally, a correction would allow to make a comparison between two different P values in a way that it is independent of sample size. Feb 7 '20 at 16:22
• You should beware of comparing p-values in the first place. They are NOT a measure of effect size, for example. If you understand what a p-value is, it's obvious that they must take sample size into account in the way that they do. Feb 8 '20 at 4:07
• An example. If I toss a coin twice and get heads twice, is that a strong indication I have a biased coin? (No, getting the same side twice happens half the time with a fair coin; it's a completely unremarkable outcome) .... But if I toss a coin 200 times and get 200 heads is that a strong indication I have a biased coin? Absolutely! Feb 8 '20 at 4:09

The effect you are observing --- that the p value decreases with a larger sample size for the same effect size --- is a feature of p values. That's just the way p values work.

In contrast, a proper effect size statistic will not increase as the sample size increases for the same effect size. It's a good practice to report effect size statistics, or some other measure of effect size (like proportions or differences in means).

For a chi-square test of association, common effect size statistics are phi and Cramer's V

For a chi-square goodness-of-fit test, there's no standard effect size statistic that I know of, except maybe just comparing observed proportions to theoretical proportions.

But also for a goodness-of-fit test we can make a modification of Cramer V statistic, taking the square root of the chi-square value divided by the sample size divided by the number of categories minus 1. See details at the documentation for this function in the rcompanion package, with the caveat that I am the author of this package, and the code below.

To demonstrate how effect sizes are independent of sample size, I'm going to create a third set of values which has the same proportions as Nobs and Nexp, but with three times the sample size.

You'll note the proportions and modified Cramer's V statistic are the same for Nobs and for Nobs3.

Nobs = c(34, 34, 41, 28)
Nexp = c(26, 44, 45, 22)
Pexp = Nexp / sum(Nexp)

chisq.test(Nobs, p = Pexp)

### Chi-squared test for given probabilities
###
### X-squared = 6.7262, df = 3, p-value = 0.08116

### Report proportions observed

Pnobs = Nobs / sum(Nobs)

Pnobs

### 0.2481752 0.2481752 0.2992701 0.2043796

Pexp

### 0.1897810 0.3211679 0.3284672 0.1605839

### Modified Cramer's V

Chi.sq    = chisq.test(x=Nobs, p=Pexp)$statistic K = length(Nobs) N = sum(Nobs) CV = sqrt(Chi.sq/N/(K-1)) names(CV) = "Modified Cramer's V" CV ### Modified Cramer's V ### 0.1279274 # # # # # # # # # # # # # # # # # Nobs3 = Nobs * 3 Nexp3 = Nexp * 3 Pexp3 = Nexp3 / sum(Nexp3) chisq.test(Nobs3, p = Pexp3) ### Chi-squared test for given probabilities ### ### X-squared = 20.179, df = 3, p-value = 0.0001559 ### Report proportions observed Pnobs3 = Nobs3 / sum(Nobs3) Pnobs3 ### 0.2481752 0.2481752 0.2992701 0.2043796 Pexp3 ### 0.1897810 0.3211679 0.3284672 0.1605839 ### Modified Cramer's V Chi.sq = chisq.test(x=Nobs3, p=Pexp3)$statistic
K         = length(Nobs3)
N         = sum(Nobs3)
CV        = sqrt(Chi.sq/N/(K-1))
names(CV) = "Modified Cramer's V"
CV

### Modified Cramer's V
###           0.1279274