Why does the chi-sq test tends to disproportionately favour p values close to 1 when there is no true difference between groups?

Question

I am playing with a simulator to look at how pre-test probability affects trial outcomes. Whilst looking at the p-values of 'control group' data it seems that they become more volatile and skewed towards 1 in the absence of a true effect.

I would have thought that if the null hypothesis is true then p-values should be evenly distributed and a histogram of them would be essentially flat. This effect seems to persist for a variable number of trial simulations, but become much more pronounced at a low n. I've included four histograms to illustrate.

1. n of 5000 and 100,000 simulations:

2. n of 500 and 100,000 simulations:

3. n of 100 and 100,000 simulations:

4. n of 5000 and 500 simulations:

MWE of the R code is:

# Variables to set
var_n = 5000        # Number in trial
var_cer = 0.2       # control event rate
var_nsim = 100      # number of simulations

# Trial run takes the control_event rate, the effect size of the intervention, and the number of trial participants, and runs a single simulation

trial_run = function(control_event_rate, effect_size, n){
  n = 0.5 * n                                                 
  
  control_alive = rbinom(1, n, control_event_rate)            # Randomly generate control group events

  treat_event_rate = control_event_rate - effect_size
  treat_alive = rbinom(1, n, treat_event_rate)                # Randomly generate treatment group events

  trial_matrix = matrix(c(control_alive, n - control_alive,   # Generate contingency table
                          treat_alive, n - treat_alive),
                        nrow = 2,
                        ncol = 2,
                        byrow  = TRUE)
  
  result = data.frame(control_alive = trial_matrix[1,1],
                      control_dead = trial_matrix[1,2],
                      treat_alive = trial_matrix[2,1],
                      treat_dead = trial_matrix[2,2],
                      p_value = chisq.test(trial_matrix)$p.value)
  result                                           
}

# Run a bunch of simulations
control_vs_control = do.call("rbind",
                             replicate(var_nsim,
                                       trial_run(control_event_rate = var_cer,   
                                                 effect_size = 0,                #trial_run called with an effect size of 0 for control vs control
                                                 n = var_n),         #number of c_c runs
                                       simplify = FALSE))

Answer 1

The p-value is uniformly distributed under the null when the sample space for the test statistic is continuous. If the sample space for the test statistic is discrete then this phenomenon does not hold, but that does not mean the test is invalid. In this instance you are approximating a discrete sampling distribution with a continuous distribution.

This is analogous to performing a test about a Bernoulli proportion. The sampling distribution of the sufficient statistic is the binomial distribution which can be referenced for performing an exact test. Over repeated experiments the p-value will not be uniformly distributed, but we can still calculate the p-value and construct an $\alpha$-level test. The $\alpha$ level will depend on the rejection region chosen within the discrete sample space. We could approximate the p-value using a normal distribution.