3
$\begingroup$

I am playing with a simulator to look at how pre-test probability affects trial outcomes. Whilst looking at the p-values of 'control group' data it seems that they become more volatile and skewed towards 1 in the absence of a true effect.

I would have thought that if the null hypothesis is true then p-values should be evenly distributed and a histogram of them would be essentially flat. This effect seems to persist for a variable number of trial simulations, but become much more pronounced at a low n. I've included four histograms to illustrate.

  1. n of 5000 and 100,000 simulations:

enter image description here

  1. n of 500 and 100,000 simulations:

enter image description here

  1. n of 100 and 100,000 simulations:

enter image description here

  1. n of 5000 and 500 simulations:

enter image description here

MWE of the R code is:

# Variables to set
var_n = 5000        # Number in trial
var_cer = 0.2       # control event rate
var_nsim = 100      # number of simulations

# Trial run takes the control_event rate, the effect size of the intervention, and the number of trial participants, and runs a single simulation

trial_run = function(control_event_rate, effect_size, n){
  n = 0.5 * n                                                 
  
  control_alive = rbinom(1, n, control_event_rate)            # Randomly generate control group events

  treat_event_rate = control_event_rate - effect_size
  treat_alive = rbinom(1, n, treat_event_rate)                # Randomly generate treatment group events

  trial_matrix = matrix(c(control_alive, n - control_alive,   # Generate contingency table
                          treat_alive, n - treat_alive),
                        nrow = 2,
                        ncol = 2,
                        byrow  = TRUE)
  
  result = data.frame(control_alive = trial_matrix[1,1],
                      control_dead = trial_matrix[1,2],
                      treat_alive = trial_matrix[2,1],
                      treat_dead = trial_matrix[2,2],
                      p_value = chisq.test(trial_matrix)$p.value)
  result                                           
}

# Run a bunch of simulations
control_vs_control = do.call("rbind",
                             replicate(var_nsim,
                                       trial_run(control_event_rate = var_cer,   
                                                 effect_size = 0,                #trial_run called with an effect size of 0 for control vs control
                                                 n = var_n),         #number of c_c runs
                                       simplify = FALSE))
$\endgroup$
3
  • 3
    $\begingroup$ The distribution is discrete and a histogram can be misleading. Look at the cdf and you may get better insights. $\endgroup$
    – Glen_b
    Dec 9, 2021 at 4:46
  • $\begingroup$ This was really helpful for visualising the relationship, thanks. $\endgroup$
    – cjdbarlow
    Dec 10, 2021 at 9:09
  • 1
    $\begingroup$ Another possibility is to use plot(table(x)) or plot(table(x)/length(x)) to see the pmf clearly but the cdf has the advantage that the relationship to uniformity is made explicit (draw in the y=x line if it's not obvious) $\endgroup$
    – Glen_b
    Dec 11, 2021 at 6:37

1 Answer 1

7
$\begingroup$

The p-value is uniformly distributed under the null when the sample space for the test statistic is continuous. If the sample space for the test statistic is discrete then this phenomenon does not hold, but that does not mean the test is invalid. In this instance you are approximating a discrete sampling distribution with a continuous distribution.

This is analogous to performing a test about a Bernoulli proportion. The sampling distribution of the sufficient statistic is the binomial distribution which can be referenced for performing an exact test. Over repeated experiments the p-value will not be uniformly distributed, but we can still calculate the p-value and construct an $\alpha$-level test. The $\alpha$ level will depend on the rejection region chosen within the discrete sample space. We could approximate the p-value using a normal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.