I have to study the relation between the gender (a nominal binary variable) with a lot (like 2000 variables more) of nominal/ordinal variables. The distribution of the genders looks like this

(gender and age): enter image description here

The problem, as you can see more or less is that in the data you have three men for each woman. So, when I try to do a contingency table I get tables like this:

enter image description here

And I can't just do a chi square I think.

Then, what can I do for study the variable dependency in this cases? When the difference between the sample of subjects is too big? and n is also very big (n>20000)

(I know the difference in this case is very obvious, but I need a statistical for see it clearly, I have to compare the gender with the 2000 other variables in R, so I can't see the tables ony by one)

Thanks you so much

  • $\begingroup$ Especially with your large sample size, be sure not to place too much emphasis on p values. You may be likely to find statistically significant results even if the effect size is small. It may be helpful to also look at effect size statistics for nominal/nominal tables (phi, Cramer's V), and nominal/ordinal tables (Freeman's theta, epsilon-squared). $\endgroup$ Dec 30, 2019 at 17:45

1 Answer 1


You don't need to have balanced frequencies in a chi-squared contingency table, you only need to make sure the counts in any one table cell are not low (i.e., <5). (e.g., Yates continuity correction). The chi-squared test is comparing row and column proportions of counts, and not absolute numbers of frequencies. This is because the expected number of counts in a cell is weighted by its column total and row total.

However, in your case, with 2000 variables being tested against gender, what you call a significant test ($P<\alpha$), where $\alpha=0.05$, will be biased from the multiple testing problem resulting in too many false positives. To overcome this, you need to adjust the level of significance by $\alpha^*=\alpha/\#tests$. So in your case for e.g. 2000 tests, the $\alpha^*=0.000025=0.05/2000$. This is called a Bonferroni (or Sidak) adjustment.

The whole premise for $\alpha^*$ is that for each test you conduct, you must consider that it's like rolling the dice in the casino, for which you have to pay. For multiple tests (2000), you have to somehow pay for each test, and the way you pay is by using $\alpha^*$ instead of $\alpha=0.05$


Please note, the Bonferroni correction to the multiple testing problem is very conservative. Thus, in practice I prefer using the Benjamini-Hochberg (1995) false discovery rate (FDR) method. In genomics and molecular biology, reviewers of papers (and advisors, lab directors) expect to see a list of genes(proteins) whose e.g. FDR=0.05. Thus the list may contain 30-500 genes, the only thing known is that 5% of them are likely false positives. Which ones specifically are false positives is unknown. If the gene list is small for FDR=5%, sometimes we have ramped up to 10-15%, but reviewers know that at this level, there is greater uncertainty and there is less specificity for differential expression.

  • 1
    $\begingroup$ Exactly that was the problem, I did the chi test and I had a lot of significant relations, so I thought that the problem was for the distribution , but now I will try changing the level of significance. But for be honest I really don`t understand this change well, I mean, if the 2000 variables are different why I have to adjust anything...if I compare they separetly, I shouldn't do anything? Thanks you a lot. $\endgroup$
    – Aibloy
    Dec 30, 2019 at 0:35
  • $\begingroup$ any time you have a table of p-values for a list of variables, you should apply Bonferroni based on #variables, which in this case: #tests = #variables. $\endgroup$
    – user32398
    Dec 30, 2019 at 14:46
  • 1
    $\begingroup$ 1. There's no law that says that you have to use a p value correction (or alpha correction) like Bonferroni just because you have multiple tests. But in that case you do have to accept that you are likely to report false positives. Also, it's important to note that a Bonferrroni correction is often rather too conservative. It's worthwhile to look in to other methods that control familywise error rate or those that control false discovery rate. $\endgroup$ Dec 30, 2019 at 17:36
  • $\begingroup$ 2. In this case, dividing a nominal alpha of 0.05 by 2000 would be unlikely to result in satisfactory results. $\endgroup$ Dec 30, 2019 at 17:36
  • 1
    $\begingroup$ The opening statement should make it clear that low expected frequencies are the biggest concern (although wanting them all to be 5 or more is often more stringent than need be). $\endgroup$
    – Nick Cox
    Jan 4, 2020 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.