# Studying the distribution of the p-value?

I have two data vectors $X=(X_1, X_2, \dots, X_n)$ and $Y=(Y_1, Y_2, \dots, Y_m)$ where $X_i \overset{iid}{\sim} P$ and $Y_i \overset{iid}{\sim} Q$, and I am performing a two-sample test (i.e KS test in the univariate case). The hypotheses are

$$H_0: P = Q\\ H_1: P \ne Q$$

I have only have a single realization of $X = (x_1, x_2, \dots, x_n)$. However I have a data generator for $Y$ (although the distribution $Q$ is still unknown), so I can obtain many samples of the form $(y_1, y_2, \dots, y_m)$.

I understand that the $p$-value is itself a random variable, and under the null hypothesis it is uniformly distributed on the interval $[0,1]$. If I only had one $X$ and one $Y$ then I can calculate only a single $p$-value (call it $p^*$) given the data, and would reject $H_0$ if $p^* < \alpha$ for some pre-specified significance level $\alpha$.

However, I have a data generator for $Y$, so I can repeat the test many times and obtain many $p$-values $\{p^*_i\}$. How can I adapt the hypothesis testing procedure to account for this fact?

Aside: I can estimate the density $Q$ and make the test parametric, however I am more interested in keeping the test non-parametric and learning how to use many $p$-values.

Since you will be using the same set of $x$ variables the resulting p-values will not be independent from each other, so even if the overall null hypothesis is true the p-values will not necessarily follow a uniform distribution.

The KS test does not have any restrictions on the size of the datasets used, so instead of generating a set of $y$s, computing a p-value, generating another set of $y$s, computing a p-value, etc. Just generate a bunch of samples of $y$ and combine them together (since they are i.i.d.) into one big set of $y$s and use that in the KS test to get a single p-value. This will give a very good picture of the shape of $Q$.