Chi-squared test simulation significance threshold

Question

I have asked question on CV here so this is a follow up. I believe this time I have the chi-squared test correct.

In this example I'm simulating o (observed frequency) by increments that deviate from the expected frequency e so to check at which point alpha=0.05 the observed is equal to observed values.

It turns out that the observed values are considered equal at pair with expected when the observed differ in % per group relative to expected by

(results without running the code)

"-2.3552092%" "-1.8722091%" "-1.6773101%"

*Conclussion: *

The above % values are relative % differences between observed and expected (by how much can the observed deviate in % terms from expected at which p.value is > 0.05).

The % results are very tiny amount, the observed values can differ by around 2% only!

*Question: * Is the chi-squared test result not overly sensitive? Meaning small relative % difference is reported being significant when comparing these groups.

*Explanation of the code (for those not using R) *

1) generate seq of observed values by to point of reaching expected values per group

2) use this seq to calc. chi-squared on every combination up to the point where observed and expected count matchs 100%.

3) find exact observed values at which p.value is >0.05.

4) calculate % relative difference of observed vs. expected per group. (you get the 3 % values per group as above.

Full simulation of the results below:

x <- data.frame(group=c(1,2,3),
            o=c(695301,154100, 224140),
            e=c(930785, 192893, 273400))


tb <- xtabs(cbind(o, e) ~ group, data=x)
tb

# visual inspection does not confirm large deviation from expected n. obs.

mosaicplot(tb, col=c(rainbow(10)[2], rainbow(10)[7])) 

# create vector nearing the expectation (convergence to expected value 
                                         by creating increments)

spl <- with(x, split(x[ ,c("o","e")], group)) 
lsim <- lapply(spl, function(x) with(x, seq(o, e, ,length.out = 1000)))  
dsim <- do.call(cbind, lsim)
o <- dsim; #rownames(o) <- "o"
e <- t(x[ ,"e", drop=FALSE])


simd <- lapply(1:nrow(o), function(i) { oi <- o[i, ,drop=FALSE]; 
               res <- rbind(oi,e); data.frame(group=1:3, t(res)) })
simul <- lapply(simd, function(x) { tb <- xtabs(cbind(V1, e) ~ group, 
                data=x); chsq <- chisq.test(t(tb)); 
                p.value <- chisq.test(t(tb))$p.value; p.value } )
simres <- round(do.call(rbind, simul),4)

plot(simres ~ 1)
lines(smooth(simres), col="red", lwd=2)
abline(v=which(simres>0.05)[1], col="blue", lwd=2)

min_accept <- simd[[which(simres>0.05)[1]]]; colnames(min_accept)[2] <- "o"  
paste0(zapsmall(with(min_accept, 100*(o - e)/e)), "%")

* Final question: * The provide answer by N. Wang is correct and confirmed by simulation.

I rescaled the observed and expected by x/100 and run the analysis/simulation. Now the values of % difference is: [1] "-20.690386%" "-16.447257%" "-14.735079%" for p.value >0.05 !

So the question is how to make the test somehow size less sensitive?

Answer 1

It looks like you have a very large sample size, with your observed/expected counts in the hundreds of thousands. As your sample size grows, smaller proportional differences become significant. If you had only 10 observed counts, an expected value of 11 would be close, well within the sampling error (it's only off by 1). If you have 100,000 counts, an expectation of 110,000 might be quite different indeed - you've sampled a bunch and are still seeing that difference which can't be explained by random sampling.