# Assessing the validity of null data generation methods for bootstrap p-values

Suppose you have a (simulated or real) sample consistent with some arbitrary $H_0$ (maybe something as simple as $\bar{x} = 0$ or something more complex involving a substantial number of parameters). Suppose as well that you have a statistic, call it $T(x)$, whose distribution is not easy to classify.

Because we can't simply compare our $T(x)$ to a known distribution, we want to generate a boostrap p-value (using parametric bootstrap). That means computing $n$ bootstrap samples consistent with $H_0$, computing $T(x_{*1}),...,T(x_{*n})$, and finding the proportion of our test statistics computed based on our bootstrap samples (recall: consistent with $H_0$) greater than the $T(x)$ based on our original data.

Now suppose generating boostrap samples based on $H_0$ is not a straightforward process. There are, potentially, numerous parameters involved, with restrictions on their values that make simulation difficult.

Because of this difficulty, suppose we end up with two competing methods of generating data under $H_0$ for our bootstrap samples. One is very simple but may make too many assumptions. The other attempts to more carefully control all of the parameters, but may be unnecessarily complex and may impose too many restrictions on the data.

My question is: how can we compare these methods of null data generation? If we are confident that our initial sample is consistent with $H_0$, can we consider the distribution of the p-values resulting from each method of null data generation for generating bootstrap p-values?

Following this discussion, for instance, can we simulate a large number of p-values for each of the two methods and simply consider which distribution of p-values is closer to uniform (again, given that the initial data we generate is consistent with $H_0$)?

In summary: for generating bootstrap p-values, we are uncertain of the best method of generating data under $H_0$. Can we consider the distribution of p-values to guide us in choosing which method is better/more consistent with our null hypothesis?

• I will formulate this into a proper answer if I don't get any responses. I discussed this with some other people and concluded that it was, if not a complete solution, at least a useful test. If one of the methods of null data generation was biased, it would have impacted the distribution of p-values. I also made power curves based on gradually increasing some of the parameters. I think the combination of these two simulations, though they do not provide an indisputable theoretical defense of one method over the other, at least show which of the methods result in a test that works. – djlid Oct 20 '17 at 13:38