# Understanding Chi Square Post hoc test results

I'm using R and comparing the proportion of Male and Female authors from various academic journals (well, I think I am). Here's a table with my raw data:

The code for it:

m<-structure(c(8278L, 2183L, 0L, 3844L, 590L, 2341L, 1659L, 422L,
0L, 899L, 137L, 662L), .Dim = c(6L, 2L), .Dimnames = structure(list(
c("PRx", "AXJ", "JAPA", "JSPRAS", "CPX", "PRX-TO"), c("Male",
"Female")), .Names = c("", "")), class = "table")


I created that table above with this code:

m<-table(data3$$Gender,data3$$Journal)


I run a chi-square test and get significant results:

chisq.test(data3$$Journal,data3$$Gender)


And when I do a chi square post hoc test in R using this code:

chisq.posthoc.test(m)


Some of the results really don't make sense to me.

For example, "PRX" has a significant value, which to me indicates that PRX is different from the other groups. But as a proportion (females are 16.69% of count for PRX), thats pretty much the same proportion as AXJ and JSPRAS (16.19% and 18% respectively) which PRX is in between. What am I not understanding correctly? It sounds like this test is showing something different than what I think it is maybe?

If I'm using the wrong test for my question, is there a better way to test it?

• The problem is that you're stuck in the mentality of $p<0.05$ = significant. When PRX has 9,937 publications, your test is overpowered. Commented Mar 4, 2021 at 17:24

First, it's not clear why you include JAPA with $$0$$ counts in both columns. What useful information does that provide?

Second, if the issue is whether there are fewer female authors than male ones, with a overall ratio of $$17,036:3779,$$ the answer is overwhelmingly Yes with no need for a statistical test. If you must have one, here is output from prop.test in R:

prop.test(c(17036, 3779), c(20815, 20815), cor=F)

2-sample test for equality of proportions
without continuity correction

data:  c(17036, 3779) out of c(20815, 20815)
X-squared = 16887, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
0.6294907 0.6443022
sample estimates:
prop 1    prop 2
0.8184482 0.1815518


Third, if the question is whether the male to female ratio differs by journal, the overall answer is Yes, on account of the tiny P-value for a test of homogeneity on TBL.

TBL = rbind(c(8278, 2183, 3844, 390, 2341),
c(1659, 422, 899, 137, 662))
TBL
[,1] [,2] [,3] [,4] [,5]
[1,] 8278 2183 3844  390 2341
[2,] 1659  422  899  137  662

chisq.test(TBL)

Pearson's Chi-squared test

data:  TBL
X-squared = 75.376, df = 4, p-value = 1.659e-15


Pearson residuals (the squares of which sum to give the chi-squared statistic) are as shown below. Looking at the residuals with the largest absolute values often points the way to interesting differences. It seems that the first two columns (Prx and AXJ) have lowest proportions of women and that the last two (CPX and AXJ) are a little nearer 50:50.

 chisq.test(TBL)\$resi
[,1]      [,2]       [,3]      [,4]      [,5]
[1,]  1.608733  1.103264 -0.6082985 -1.989678 -2.355970
[2,] -3.415697 -2.342474  1.2915527  4.224527  5.002247


It might be worthwhile to do a chi-squared test that omits column 3 (where the ratio is closer to the overall ratio). The disparity is also overwhelmingly significant with a P-value near $$0.$$

chisq.test(TBL[, c(1,2,4,5)])

Pearson's Chi-squared test

data:  TBL[, c(1, 2, 4, 5)]
X-squared = 73.482, df = 3, p-value = 7.664e-16


What needs further examination and (perhaps) formal testing depends on your purpose for doing this study. Are you surprised that there are more male than female authors? Did you expect that the degree of disparity would differ markedly by journal or journal type? An unmotivated list of tiny P-values is not going to lead to interesting or useful conclusions.

• The overall question was whether one journal (or several) had appreciably different ratios of female to male authors compared to the other journals (ad which one it was). Yes, its fairly obvious that as a whole there are more male authors. I think Adam0's comment answered my question of why the post-hoc test confused me. JAPA was left in there because it was a category in my real data that we didn't find any examples for. Commented Mar 5, 2021 at 13:35
• According to first of my chi-sq tests, answer to that is Yes. Commented Mar 5, 2021 at 17:09
• Does that answer which one it was? If so, what is the point of a chi square post hoc test? Commented Mar 5, 2021 at 17:21