# References: t-test and Chi-squared test can be conducted with unequal sample sizes

I have two independent groups: sample size of group 1 is 24000, and sample size of group 2 is 246. With the drastically different sample sizes in the two group, I need to use the Student’s t-test or the Welch’s t-test or the Wilcoxon rank-sum (Mann-Whitney U) test as appropriate for continuous variables, and use the Chi-squared test or the Fisher's exact test as appropriate for categorical variables to check the significant differences between group 1 and group 2.

I have studied some blogs or chat forum posts including the answers in "Cross Validated", and learned that I can conduct these statistical tests for comparing two groups with very different sample sizes.

However, I still have two confusions:

1. These statistical tests have dealt with the situation that hypothesis testing is prone to significance due to very different sample sizes, haven't they? Or, I need do something else to make the results reliable.

2. I can't find references instead of blogs or chat forum posts to support that these statistical tests are fine and persuade the reviewers.

Can you help? Anything tips can help a lot.

I have spend two days to seek references and I failed.

• The pooled 2-sample t test assumes equal population variances. Particularly when sample sizes are very different the pooled test can give incorrect P-values if this assumption is not net. It is a good idea always to use the Welch 2-sample test unless you have solid prior evidence that population variances are nearly equal. // As long as all of the expected counts in a chi-squared test exceed 5, the distribution of the null distribution should have a chi-squared dist'n. In R, and many other statistical programs, you will get a warning msg if some expected counts are too small. Commented May 5, 2022 at 15:21
• You ask for references. If that's to help understand what's important in practice, I hope my Answer helps. If you need to refer to journal articles for discussions with reviewers, then a good strategy is to look at Wikipedia pages on 'Welch t test', 'Chi-squared simulated P-value', 'Fisher Exact Test', "Yates continuity correction', etc. Then look at the Wikipedia bibliographies. Commented May 5, 2022 at 18:30
• Thank you very much! I have conducted homogeneity test of variance for continuous variables and when population variances were unequal I used the Welch 2-sample test otherwise the pooled 2-sample t test. For categorical variables, as long as one of the expected counts in a chi-squared test was less than or equal to 5, I used the Fisher's exact test otherwise the Chi-squared test. Commented May 6, 2022 at 7:30
• The suggestion about the Wikipedia bibliographies you gave is helpful! Thanks again! You've given me another perspective, and I am doing the work on it. Commented May 6, 2022 at 7:35

Many intermediate-level applied statistical texts have the warnings mentioned in my comment. (One example, among many, is the text by Ott & Longnecker.)

Use Welch, not pooled t test. Here is an example of the very bad behavior of the pooled 2-sample t test of $$H_0: \mu_1 = \mu_2$$ against $$H_a: \mu_1 \ne \mu_2$$ for unbalances sample sizes when the smaller sample is from a population with a larger variance than for the larger sample. Suppose $$n_1 = 20, n_2 = 80;$$ $$\mu_1 = \mu_2 = 50;$$ $$\sigma_1^2 = 10^2 = 100,$$ $$\sigma_2^2 = 3^2 = 9.$$ Then the pooled test (paramater var.eq=T) that is intended to be at the 5% level actually has a Type I error about 27%. That can result in routine 'false discovery'. Simulation in R:

set.seed(2022)
pv = replicate(10^5, t.test(rnorm(20, 50, 10), rnorm(80, 50, 3), var.eq=T)$p.val) mean(pv <= 0.05) [1] 0.26586  If we use the Welch 2-sample t test instead, then the significance level is very nearly correct. set.seed(505) pv = replicate(10^5, t.test(rnorm(20, 50, 10), rnorm(80, 50, 3))$p.val)
mean(pv <= 0.05)
[1] 0.05065


Simulated P-value for chi-squared test with small expected values.

Suppose we have the following table of counts.

TBL
[,1] [,2] [,3]
[1,]  102  304    2
[2,]  205  418    5


A chi-squared test of independence of row and column categorical variables, using this table of observed counts, we get a P-value that would indicate rejection of the null hypothesis (at the 5% level), if correct. But a warning message indicates that the P-value should not be trusted.

chisq.test(TBL)

Pearson's Chi-squared test

data:  TBL
X-squared = 7.461, df = 2, p-value = 0.02398

Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect


The difficulty is that the expected counts in the third column are both below $$5$$ on account of the sparse data in that column.

chisq.test(TBL)\$exp
[,1]     [,2]     [,3]
[1,] 120.9035 284.3398 2.756757
[2,] 186.0965 437.6602 4.243243
Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect


As implemented in R, one can simulate a more reliable P-value, significant at the 5% level.

chisq.test(TBL, sim=T)

Pearson's Chi-squared test
with simulated p-value
(based on 2000 replicates)

data:  TBL
X-squared = 7.461, df = NA, p-value = 0.02499


Moreover, some statistical software programs (including R) will compute Fisher's Exact Test for tables of counts somewhat larger than $$2\times 2.$$ (Very large tables that require a lot of memory and time may not work on 'ordinary' computers.)

Here is Fisher's exact test from R for the current TBL.

fisher.test(TBL)

Fisher's Exact Test for Count Data

data:  TBL
p-value = 0.01881
alternative hypothesis: two.sided

• The explanation is clear and precise. I thought I have answer for No.1 confusion. I am try solve my second problem. I would like to extend my sincerest gratitude to you, BruceET. It is the first time for me to seek help here and I am really glad I did. Commented May 6, 2022 at 7:45