I just finished reading the following article by Berger & Berry (1988) in which they explain how subjectivity enters statistical analyses. One of their examples concerns a clinical trial with either one or two stages (pp. 164 in the article). The scenario looks as follows:
- we have n matched pairs of subjects.
- each member of a pair receives vitamin C and a placebo, respectively
- we want to assess whether the subject receiving vitamin C or the subject receiving the placebo exhibits greater relief from cold symptoms (it's a purely fictional experiment, so don't hang me up on the medical aspects)
- our null hypothesis is that there is no difference between vitamin C and the placebo (i.e. p(vitamin C helps) = p(placebo helps) = 1/2; hence, we're dealing with a binomial distribution
1) In the first case, there is only one stage including n=17 pairs of patients. If the number of pairs with preference for vitamin C (i.e. the number of pairs for which vitamin C seemed to help) is an element of [0,1,2,3,4,13,14,15,16,17], i.e. if it is smaller than 5 or bigger than 12, the null-hypothesis shall be rejected. The summed probability of getting a number smaller than 5 or bigger than 12 under the null hypothesis is 0.049
2) In the second case, we are dealing with two stages. In the first stage, we have n=17 pairs of patient. Again, we reject the null hypothesis if the number of pairs with a preference for vitamin C is smaller than 5 or bigger than 12. However, if we are not able to reject the null hypothesis based on this criterion, we add a second stage to our trial by looking at an additional 27 pairs, for a total of 44 pairs, concluding that there is sufficient evidence against H if the total number of preferences for vitamin C is less than 16 or more than 28. The summed probability of these events is again 0.049.
Now, Berger & Berry argue that in the case that we have 13 preferences for vitamin C out of 17 pairs the p-value is different in the first case with only one planned and conducted stage compared to the second stage with two planned, but only one conducted stage.
Their explanation: "To see this, recall the basic process for arriving at a P-value. One assumes that H is true, calculates the probability of the set of possible data which would cast as much or more doubt on H than the observed data, and claims significant evidence against H if this probability is small enough. The set R* of more extreme observations in the two-stage design equals the set R of more extreme observations for the one-stage design (n=17) plus the more extreme observations at the second stage (n=44). Since R is contained in R*, it is clear that R* has a larger probability and hence is less "significant". The probability of hitting a region outside the significance threshold after 17 observations or, failing that, after 44 observations, turn out to be 0.085"
I understand the argument that R* comprises R, so R* cannot be smaller than R. I couldn't figure out, however, how the authors got the number 0.085. My approach would have been to add up the probability of getting a significant result in the first stage (0.049) and the probability of getting a significant result in the second stage (0.049) times the probability of actually getting to the second stage (1-0.049). But this gives me: 0.049 + ((1-0.049)*0.049) = 0.095599 ≠ 0.085.
Another approach was to take into consideration that in order to get to the second stage, the number of pairs preferring vitamin C in the first stage had do lie between 5 and 12 (otherwise we would have rejected the null hypothesis and stopped the experiment). Hence, the number of pairs preferring vitamin C can only lie between 5 and 39. But this information didn't help me either to get to the right result.
Could somebody explain how I can get the type I error of 0.089 for the above-described scenario?