0
$\begingroup$

I have two independent groups: sample size of group 1 is 24000, and sample size of group 2 is 246. With the drastically different sample sizes in the two group, I need to use the Student’s t-test or the Welch’s t-test or the Wilcoxon rank-sum (Mann-Whitney U) test as appropriate for continuous variables, and use the Chi-squared test or the Fisher's exact test as appropriate for categorical variables to check the significant differences between group 1 and group 2.

I have studied some blogs or chat forum posts including the answers in "Cross Validated", and learned that I can conduct these statistical tests for comparing two groups with very different sample sizes.

However, I still have two confusions:

  1. These statistical tests have dealt with the situation that hypothesis testing is prone to significance due to very different sample sizes, haven't they? Or, I need do something else to make the results reliable.

  2. I can't find references instead of blogs or chat forum posts to support that these statistical tests are fine and persuade the reviewers.

Can you help? Anything tips can help a lot.

I have spend two days to seek references and I failed.

$\endgroup$
4
  • 1
    $\begingroup$ The pooled 2-sample t test assumes equal population variances. Particularly when sample sizes are very different the pooled test can give incorrect P-values if this assumption is not net. It is a good idea always to use the Welch 2-sample test unless you have solid prior evidence that population variances are nearly equal. // As long as all of the expected counts in a chi-squared test exceed 5, the distribution of the null distribution should have a chi-squared dist'n. In R, and many other statistical programs, you will get a warning msg if some expected counts are too small. $\endgroup$
    – BruceET
    May 5, 2022 at 15:21
  • $\begingroup$ You ask for references. If that's to help understand what's important in practice, I hope my Answer helps. If you need to refer to journal articles for discussions with reviewers, then a good strategy is to look at Wikipedia pages on 'Welch t test', 'Chi-squared simulated P-value', 'Fisher Exact Test', "Yates continuity correction', etc. Then look at the Wikipedia bibliographies. $\endgroup$
    – BruceET
    May 5, 2022 at 18:30
  • $\begingroup$ Thank you very much! I have conducted homogeneity test of variance for continuous variables and when population variances were unequal I used the Welch 2-sample test otherwise the pooled 2-sample t test. For categorical variables, as long as one of the expected counts in a chi-squared test was less than or equal to 5, I used the Fisher's exact test otherwise the Chi-squared test. $\endgroup$
    – Xin Xu
    May 6, 2022 at 7:30
  • 1
    $\begingroup$ The suggestion about the Wikipedia bibliographies you gave is helpful! Thanks again! You've given me another perspective, and I am doing the work on it. $\endgroup$
    – Xin Xu
    May 6, 2022 at 7:35

1 Answer 1

1
$\begingroup$

Many intermediate-level applied statistical texts have the warnings mentioned in my comment. (One example, among many, is the text by Ott & Longnecker.)

Use Welch, not pooled t test. Here is an example of the very bad behavior of the pooled 2-sample t test of $H_0: \mu_1 = \mu_2$ against $H_a: \mu_1 \ne \mu_2$ for unbalances sample sizes when the smaller sample is from a population with a larger variance than for the larger sample. Suppose $n_1 = 20, n_2 = 80;$ $\mu_1 = \mu_2 = 50;$ $\sigma_1^2 = 10^2 = 100,$ $\sigma_2^2 = 3^2 = 9.$ Then the pooled test (paramater var.eq=T) that is intended to be at the 5% level actually has a Type I error about 27%. That can result in routine 'false discovery'. Simulation in R:

set.seed(2022)
pv = replicate(10^5, t.test(rnorm(20, 50, 10), rnorm(80, 50, 3), var.eq=T)$p.val)
mean(pv <= 0.05)
[1] 0.26586

If we use the Welch 2-sample t test instead, then the significance level is very nearly correct.

set.seed(505)
pv = replicate(10^5, t.test(rnorm(20, 50, 10), rnorm(80, 50, 3))$p.val)
mean(pv <= 0.05)
[1] 0.05065

Simulated P-value for chi-squared test with small expected values.

Suppose we have the following table of counts.

TBL
     [,1] [,2] [,3]
[1,]  102  304    2
[2,]  205  418    5

A chi-squared test of independence of row and column categorical variables, using this table of observed counts, we get a P-value that would indicate rejection of the null hypothesis (at the 5% level), if correct. But a warning message indicates that the P-value should not be trusted.

chisq.test(TBL)

    Pearson's Chi-squared test

data:  TBL
X-squared = 7.461, df = 2, p-value = 0.02398

Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect

The difficulty is that the expected counts in the third column are both below $5$ on account of the sparse data in that column.

chisq.test(TBL)$exp
         [,1]     [,2]     [,3]
[1,] 120.9035 284.3398 2.756757
[2,] 186.0965 437.6602 4.243243
Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect

As implemented in R, one can simulate a more reliable P-value, significant at the 5% level.

chisq.test(TBL, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TBL
X-squared = 7.461, df = NA, p-value = 0.02499

Moreover, some statistical software programs (including R) will compute Fisher's Exact Test for tables of counts somewhat larger than $2\times 2.$ (Very large tables that require a lot of memory and time may not work on 'ordinary' computers.)

Here is Fisher's exact test from R for the current TBL.

fisher.test(TBL)

        Fisher's Exact Test for Count Data

data:  TBL
p-value = 0.01881
alternative hypothesis: two.sided
$\endgroup$
1
  • $\begingroup$ The explanation is clear and precise. I thought I have answer for No.1 confusion. I am try solve my second problem. I would like to extend my sincerest gratitude to you, BruceET. It is the first time for me to seek help here and I am really glad I did. $\endgroup$
    – Xin Xu
    May 6, 2022 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.