3
$\begingroup$

I have two data vectors $X=(X_1, X_2, \dots, X_n)$ and $Y=(Y_1, Y_2, \dots, Y_m)$ where $X_i \overset{iid}{\sim} P$ and $Y_i \overset{iid}{\sim} Q$, and I am performing a two-sample test (i.e KS test in the univariate case). The hypotheses are

$$ H_0: P = Q\\ H_1: P \ne Q $$

I have only have a single realization of $X = (x_1, x_2, \dots, x_n)$. However I have a data generator for $Y$ (although the distribution $Q$ is still unknown), so I can obtain many samples of the form $(y_1, y_2, \dots, y_m)$.

I understand that the $p$-value is itself a random variable, and under the null hypothesis it is uniformly distributed on the interval $[0,1]$. If I only had one $X$ and one $Y$ then I can calculate only a single $p$-value (call it $p^*$) given the data, and would reject $H_0$ if $p^* < \alpha$ for some pre-specified significance level $\alpha$.

However, I have a data generator for $Y$, so I can repeat the test many times and obtain many $p$-values $\{p^*_i\}$. How can I adapt the hypothesis testing procedure to account for this fact?

Aside: I can estimate the density $Q$ and make the test parametric, however I am more interested in keeping the test non-parametric and learning how to use many $p$-values.

$\endgroup$
3
$\begingroup$

Since you will be using the same set of $x$ variables the resulting p-values will not be independent from each other, so even if the overall null hypothesis is true the p-values will not necessarily follow a uniform distribution.

The KS test does not have any restrictions on the size of the datasets used, so instead of generating a set of $y$s, computing a p-value, generating another set of $y$s, computing a p-value, etc. Just generate a bunch of samples of $y$ and combine them together (since they are i.i.d.) into one big set of $y$s and use that in the KS test to get a single p-value. This will give a very good picture of the shape of $Q$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.