Many intermediate-level applied statistical texts have the warnings mentioned in my comment. (One example, among many, is the text by Ott & Longnecker.)
Use Welch, not pooled t test. Here is an example of the very bad behavior of the pooled 2-sample t test of $H_0: \mu_1 = \mu_2$ against $H_a: \mu_1 \ne \mu_2$ for unbalances sample sizes when the smaller sample is from a population with a larger variance than for the larger sample. Suppose $n_1 = 20, n_2 = 80;$ $\mu_1 = \mu_2 = 50;$ $\sigma_1^2 = 10^2 = 100,$ $\sigma_2^2 = 3^2 = 9.$
Then the pooled test (paramater var.eq=T) that is intended
to be at the 5% level actually has a Type I error about 27%. That
can result in routine 'false discovery'. Simulation in R:
set.seed(2022)
pv = replicate(10^5, t.test(rnorm(20, 50, 10), rnorm(80, 50, 3), var.eq=T)$p.val)
mean(pv <= 0.05)
[1] 0.26586
If we use the Welch 2-sample t test instead, then the significance
level is very nearly correct.
set.seed(505)
pv = replicate(10^5, t.test(rnorm(20, 50, 10), rnorm(80, 50, 3))$p.val)
mean(pv <= 0.05)
[1] 0.05065
Simulated P-value for chi-squared test with small expected values.
Suppose we have the following table of counts.
TBL
[,1] [,2] [,3]
[1,] 102 304 2
[2,] 205 418 5
A chi-squared test of independence of row and column categorical
variables, using this table of observed counts, we get a P-value that would indicate rejection of the null
hypothesis (at the 5% level), if correct. But a warning message indicates that the P-value should not be trusted.
chisq.test(TBL)
Pearson's Chi-squared test
data: TBL
X-squared = 7.461, df = 2, p-value = 0.02398
Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect
The difficulty is that the expected counts in the third column are
both below $5$ on account of the sparse data in that column.
chisq.test(TBL)$exp
[,1] [,2] [,3]
[1,] 120.9035 284.3398 2.756757
[2,] 186.0965 437.6602 4.243243
Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect
As implemented in R, one can simulate a more reliable P-value, significant at the 5% level.
chisq.test(TBL, sim=T)
Pearson's Chi-squared test
with simulated p-value
(based on 2000 replicates)
data: TBL
X-squared = 7.461, df = NA, p-value = 0.02499
Moreover, some statistical software programs (including R) will
compute Fisher's Exact Test for tables of counts somewhat larger than $2\times 2.$ (Very large tables that require a lot of memory and time
may not work on 'ordinary' computers.)
Here is Fisher's exact test from R for the current TBL.
fisher.test(TBL)
Fisher's Exact Test for Count Data
data: TBL
p-value = 0.01881
alternative hypothesis: two.sided
