I'm considering the number of observed 'successes' $x_1$ and $x_2$ from $n_1$ and $n_2$ independent trials respectively. These counts can be considered realisations of random variables $X_1, X_2$ such that $X_1 \sim Bi(n_1, \pi_1), X_2 \sim Bi(n_2, \pi_2)$ and $\pi_1,\pi_2$ are the proportion of “successes” in the true population.

I would like to determine the probability of observing equal or more extreme than $x_1,x_2$ given that in the true population $\pi_2 \ge \pi_1$". The null and alternative hypothesis are therefore $$H_0: \pi_2 - \pi_1 \ge 0$$ $$H_1: \pi_2 - \pi_1 \lt 0$$

and I want the probability of observing counts which are equal or more extreme than those observed given the null hypothesis is true, i.e. $$ P(X_1 \ge x_1 \wedge X_2 \le x_2 | \pi_2 - \pi_1 \ge 0) $$

One way to solve this probability without knowing the nuisance parameter $\pi_{i}$ is using the supremum approach suggested by Barnard (1945). Given some test statistic, $T(X_1,X_2)$, the probability of observing $(x_1,x_2)$ under the null as $$ P = \max_{0 \le \pi_1\le \pi_2 \le 1}\sum\limits_{T(X_1,X_2) \ge T(x_1,x_2)} P(X_1,X_2|\pi_1,\pi_2) $$ where $T(X_1,X_2)>T(x_1,x_2)$ is $X_1 \ge x_1 \wedge X_2 \le x_2 $. In this case, $P(X_1,X_2|\pi_1,\pi_2)$ is just a product of binomials.


I'm concerned that the "test statistic" suggested above does not really fit into the scheme of $T(X_1,X_2)>T(x_1,x_2)$ as the conjunction of inequalities can't be separated into individual scores. Instead, I'm sort of doing two tests on each table and combining them. Concerningly, Q-Q plot for counts generated from random trials (below) shows that the resulting probabilities from this test are non-uniform and overly liberal, whereas using a Wald test as the test statistic appears to follow the expected uniform distribution reasonably well. enter image description here

I'm therefore concerned that by abusing my choice of test statistic, I'm not generating "correct" (uniform) p-value but I'm not sure.

Is my use of $X_1 \ge x_1 \wedge X_2 \le x_2 $ as a test statistic reasonable?

If it is, I'd be keen to hear why the use of this test leads to apparently liberal $p$-values and whether its similar to anything in the literature.

And if its not reasonable, I'd be keen to hear why and what a suitable alternative would be that still.

  • $\begingroup$ Can you rephrase your question to clearly specify what you want to test? What is your null hypothesis and what is your alternative hypothesis? $\endgroup$ – Peter Oct 23 '14 at 14:47
  • $\begingroup$ @Peter: I've tried to phrase hypotheses clearly and made the whole thing more succinct. Please let me know if the question remains unclear. I find it a struggle to phrase these types of problems easily :) $\endgroup$ – Zkk Oct 24 '14 at 3:22
  • $\begingroup$ You need to look into theory of comparing 2 proportions. See page 4-7 of this document. $\endgroup$ – Peter Oct 24 '14 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.