3
$\begingroup$

Suppose $T$ is a Bernoulli random variable, such that $T = 1$ with probability $s$, $s$ is not known.

I am interested in checking hypothesis $H_1$ that $s \neq 1/2$ ($H_0$ is that $s = 1/2$). One way to do that is to get some 100K of realizations of $T$, and run the binomial test, check the p-value etc. However, sometimes $s$ is pretty far from $1/2$, and in that case I want to stop my experiment as early as possible once I've got enough evidence to reject $H_0$ (low $s$ means that the experiment participants are 'suffering', and generally I want to avoid that; high $s$ means that I can improve life for everyone and generally I want to do that asap).

My idea was to deploy a sequential testing procedure: I split the experiment in 5 equally sized stages (e.g. 20K realizations each) and after each stage I calculate p-values, compare to the predefined thresholds, and decide to stop the experiment or to continue. This idea is inspired by O'Brien & Fleming procedure (O'Brien & Fleming, Biometrics '79).

There exist quite extensive literature on sequential testing, R package gsDesign, etc. However, as the topic is mostly studied in the medical context, usually a control/test split is assumed. Thus I failed to find a guidance how to design the sequential experiment protocol for my set-up: given pre-fixed values for Type I and II errors, the number of stops, I want to find p-value thresholds (or other stopping criteria) for each of the stops.

Could anyone provide me with a relevant link?

$\endgroup$
2
$\begingroup$

It might be that the group sequential testing approach is overkill for your application. That is most useful when it is costly to perform interim analyses. If you have very little cost to analysis, then you can just do sequential testing.

There are many references to cover the theory of sequential testing. Your case is usually covered as an example along with the normal distribution. (For example, see II.3 of (Siegmund, 1985)).

There is plenty of literature on acceptance sampling and process monitoring that also bears on this problem if you need to perform batch sampling.

However, for sequential testing, the original paper on the sequential probability ratio test by (Wald, 1945) is very readable and available online. Your case is covered in Section 5.2.

To apply the procedure, you will need to specify a probability that is good enough to accept ( $p_0$), and a probability that is bad enough to reject ($p_1$), in addition to your usual error rates $\alpha$ and $\beta$. Then, you monitor the number of bad outcomes.

Let $d_m$ be the number of bad outcomes at the $m$th trial. Then you accept $H_0$ if $d_m \le A_m$, accept $H_1$ if $d_m \ge R_m$, and continue sampling if $A_m < d_m < R_m$.

$A_m$ is called the acceptance number and $R_m$ is called the rejection number. They are calculated as $$A_m = \frac{ \log \frac{\beta}{1-\alpha} } { \log \frac{p_1}{p_0} - \log \frac{1-p_1}{1-p_0} } + m \frac{ \log \frac{1-p_0}{1-p_1} } { \log \frac{p_1}{p_0} - \log \frac{1-p_1}{1-p_0} } $$ and $$R_m = \frac{ \log \frac{1-\beta}{\alpha} } { \log \frac{p_1}{p_0} - \log \frac{1-p_1}{1-p_0} } + m \frac{ \log \frac{1-p_0}{1-p_1} } { \log \frac{p_1}{p_0} - \log \frac{1-p_1}{1-p_0} }. $$

There is plenty of theory there if you want, including certainty of stopping, expected sample sizes, and efficiency. This formulation does not incorporate economic costs.

References:

Siegmund, D. (1985) Sequential Analysis: Tests and Confidence Intervals. Springer-Verlag.

Wald, A. (1945). Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16(2), 117-186.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you! I already started looking into MaxSPRT procedures, but it looks like I can start with even simpler approach. Could you also clarify the distinction between 'group sequential testing' vs 'sequential testing' vs 'batch sampling'? E.g., if I run a sequential testing procedure on a data that arrives in groups, what can go wrong? $\endgroup$ – bijey Dec 7 '14 at 9:14
  • 1
    $\begingroup$ @bijey I might be using the terminology incorrectly, but sequential testing usually means as the data arrive individually. Group sequential tests I think means collecting sequential data for a while, then performing statistical analysis on the whole data set to that point. My use of "batch sampling" was a little loose. For issues about sequentially testing data that arrives in groups... not sure! I could imagine cases where it shouldn't make any difference, and vice versa. $\endgroup$ – jvbraun Dec 7 '14 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.