# Why does the chi-sq test tends to disproportionately favour p values close to 1 when there is no true difference between groups?

I am playing with a simulator to look at how pre-test probability affects trial outcomes. Whilst looking at the p-values of 'control group' data it seems that they become more volatile and skewed towards 1 in the absence of a true effect.

I would have thought that if the null hypothesis is true then p-values should be evenly distributed and a histogram of them would be essentially flat. This effect seems to persist for a variable number of trial simulations, but become much more pronounced at a low n. I've included four histograms to illustrate.

1. n of 5000 and 100,000 simulations:

1. n of 500 and 100,000 simulations:

1. n of 100 and 100,000 simulations:

1. n of 5000 and 500 simulations:

MWE of the R code is:

# Variables to set
var_n = 5000        # Number in trial
var_cer = 0.2       # control event rate
var_nsim = 100      # number of simulations

# Trial run takes the control_event rate, the effect size of the intervention, and the number of trial participants, and runs a single simulation

trial_run = function(control_event_rate, effect_size, n){
n = 0.5 * n

control_alive = rbinom(1, n, control_event_rate)            # Randomly generate control group events

treat_event_rate = control_event_rate - effect_size
treat_alive = rbinom(1, n, treat_event_rate)                # Randomly generate treatment group events

trial_matrix = matrix(c(control_alive, n - control_alive,   # Generate contingency table
treat_alive, n - treat_alive),
nrow = 2,
ncol = 2,
byrow  = TRUE)

result = data.frame(control_alive = trial_matrix[1,1],
treat_alive = trial_matrix[2,1],
p_value = chisq.test(trial_matrix)\$p.value)
result
}

# Run a bunch of simulations
control_vs_control = do.call("rbind",
replicate(var_nsim,
trial_run(control_event_rate = var_cer,
effect_size = 0,                #trial_run called with an effect size of 0 for control vs control
n = var_n),         #number of c_c runs
simplify = FALSE))

• The distribution is discrete and a histogram can be misleading. Look at the cdf and you may get better insights. Commented Dec 9, 2021 at 4:46
• This was really helpful for visualising the relationship, thanks. Commented Dec 10, 2021 at 9:09
• Another possibility is to use plot(table(x)) or plot(table(x)/length(x)) to see the pmf clearly but the cdf has the advantage that the relationship to uniformity is made explicit (draw in the y=x line if it's not obvious) Commented Dec 11, 2021 at 6:37

This is analogous to performing a test about a Bernoulli proportion. The sampling distribution of the sufficient statistic is the binomial distribution which can be referenced for performing an exact test. Over repeated experiments the p-value will not be uniformly distributed, but we can still calculate the p-value and construct an $$\alpha$$-level test. The $$\alpha$$ level will depend on the rejection region chosen within the discrete sample space. We could approximate the p-value using a normal distribution.