Is a chi-squared test valid to test whether a parameter change affects the distribution in two independent categories? I have performed an experiment in which mice either respond, or do not respond, and the mice are either mutants or wild types (i.e. non-mutants).
I would like to test whether the proportion of responders is different between the mutant and the wild type mice.
I am struggling to decide whether a chi-squared test is appropriate here, since I am not exactly testing whether the populations are independent of one another (I don't think) but rather whether the two populations have statistically different distributions.
Of course, if chi-squared is not valid, I welcome suggestions as to what is.
 A: If you need to make a distinction between
a response rate of $.20$ and $.25,$ of course
you will need more mice in each group than if
you want to distinguish between $.20$ and $.45.$
For example, suppose the response rate for
mutant mice is $.20$ and $.47$ for wild mice.
Then for 40 mice in each group, you might get
the fictitious counts below.
set.seed(112)
muta = rbinom(1, 40, .2)
wild = rbinom(1, 40, .47)
yes = c(muta, wild)
tot = c(40,40)
no = tot - yes
TAB = rbind(yes, no);  TAB
TAB
     [,1] [,2]
 yes    5   19
 no    35   21

Then chisq.test in R gives a P-value
$0.0006 < 0.05 = 5\%,$ so the difference in
Yes rates is significantly different between
mutant and wild mice at the 5% level (also at the 0.1% level).
chisq.test(TAB, cor=F)

        Pearson's Chi-squared test

data:  TAB
X-squared = 11.667, df = 1, p-value = 0.0006363

Fisher's exact test also finds a significant
difference at the 5% level. You might use this
test--especially, in case counts are too small to get a
reliable P-value from the chi-squared test.
[However, if the chi-squared test shows a warning message about small counts, you might get a more useful (simulated) P-value using parameter sim=T.]
fisher.test(TAB)

        Fisher's Exact Test for Count Data

data:  TAB
p-value = 0.001237
alternative hypothesis: 
 true odds ratio is not equal to 1
95 percent confidence interval:
 0.04092359 0.53377166
sample estimates:
odds ratio 
 0.1617825 

In R, prop.test is essentially equivalent
to the chi-squared test using a $2\times 2$ table; without 'continuity correction' the P-value is the same.
(Notice that the syntax is a little different and the output may be more convenient for some purposes. Also, you can use parameter
alt="greater" or alt="less" to do a one-sided test.)
prop.test(yes, tot, cor=F)

        2-sample test for 
        equality of proportions 
        without continuity correction

data:  yes out of tot
X-squared = 11.667, df = 1, p-value = 0.0006363
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.5356154 -0.1643846
sample estimates:
prop 1 prop 2 
 0.125  0.475 

If you are wondering ahead of time whether using
20 mice per group (instead of 40) would give good enough
power to detect a difference between
proportions $.20$ and $.45$ of Yes's, then
you might do a simulation as below. The answer
is that with 20 mice in each group you'd
detect such a difference at the 5% level less than a third of
the time.
set.seed(2022)
n = 20;  m = 10^5;  pv = numeric(m)
for (i in 1:m) {
 muta = rbinom(1, n, .2)
 wild = rbinom(1, n, .45)
 yes = c(muta, wild)
 tot = c(n,n)
 no = tot - yes
 TAB = rbind(yes, no);  TAB
 pv[i] = fisher.test(TAB)$p.val
 }
mean(pv <= .05)
[1] 0.29122

With sample sizes of 65, Fisher's exact test would detect this difference
about 80% of the time.
