# Chi-squared test simulation significance threshold

I have asked question on CV here so this is a follow up. I believe this time I have the chi-squared test correct.

In this example I'm simulating o (observed frequency) by increments that deviate from the expected frequency e so to check at which point alpha=0.05 the observed is equal to observed values.

It turns out that the observed values are considered equal at pair with expected when the observed differ in % per group relative to expected by

(results without running the code)

# "-2.3552092%" "-1.8722091%" "-1.6773101%"

*Conclussion: *

The above % values are relative % differences between observed and expected (by how much can the observed deviate in % terms from expected at which p.value is > 0.05).

The % results are very tiny amount, the observed values can differ by around 2% only!

*Question: * Is the chi-squared test result not overly sensitive? Meaning small relative % difference is reported being significant when comparing these groups.

*Explanation of the code (for those not using R) *

1) generate seq of observed values by to point of reaching expected values per group

2) use this seq to calc. chi-squared on every combination up to the point where observed and expected count matchs 100%.

3) find exact observed values at which p.value is >0.05.

4) calculate % relative difference of observed vs. expected per group. (you get the 3 % values per group as above.

Full simulation of the results below:

x <- data.frame(group=c(1,2,3),
o=c(695301,154100, 224140),
e=c(930785, 192893, 273400))

tb <- xtabs(cbind(o, e) ~ group, data=x)
tb

# visual inspection does not confirm large deviation from expected n. obs.

mosaicplot(tb, col=c(rainbow(10), rainbow(10)))

# create vector nearing the expectation (convergence to expected value
by creating increments)

spl <- with(x, split(x[ ,c("o","e")], group))
lsim <- lapply(spl, function(x) with(x, seq(o, e, ,length.out = 1000)))
dsim <- do.call(cbind, lsim)
o <- dsim; #rownames(o) <- "o"
e <- t(x[ ,"e", drop=FALSE])

simd <- lapply(1:nrow(o), function(i) { oi <- o[i, ,drop=FALSE];
res <- rbind(oi,e); data.frame(group=1:3, t(res)) })
simul <- lapply(simd, function(x) { tb <- xtabs(cbind(V1, e) ~ group,
data=x); chsq <- chisq.test(t(tb));
p.value <- chisq.test(t(tb))\$p.value; p.value } )
simres <- round(do.call(rbind, simul),4)

plot(simres ~ 1)
lines(smooth(simres), col="red", lwd=2)
abline(v=which(simres>0.05), col="blue", lwd=2)

min_accept <- simd[[which(simres>0.05)]]; colnames(min_accept) <- "o"
paste0(zapsmall(with(min_accept, 100*(o - e)/e)), "%")


* Final question: * The provide answer by N. Wang is correct and confirmed by simulation.

I rescaled the observed and expected by x/100 and run the analysis/simulation. Now the values of % difference is:  "-20.690386%" "-16.447257%" "-14.735079%" for p.value >0.05 !

So the question is how to make the test somehow size less sensitive?

• I'm sorry. I have now provided direct question and wrote concluding remark. Oct 5, 2018 at 12:50
• I have now also provided explanation of the r code what it does. Oct 5, 2018 at 13:09

• (+1) thank you looking into this. You are right! I rescaled the observed and expected by x/100 and run the analysis/simulation. Now the values of % difference is:  "-20.690386%" "-16.447257%" "-14.735079%" for p.value >0.05 ! Oct 5, 2018 at 13:17