24
$\begingroup$

An interim analysis is an analysis of the data at one or more time points prior the official close of the study with the intention of, e.g., possibly terminating the study early.

According to Piantadosi, S. (Clinical trials - a methodologic perspective): "The estimate of a treatment effect will be biased when a trial is terminated at an early stage. The earlier the decision, the larger the bias."

Can you explain me this claim. I can easily understand that the accuracy is going to be affected, but the claim about the bias is not obvious to me...

Improve this question
$\endgroup$
1
  • $\begingroup$ I think this is a perfect question to "flesh out" the differences between Bayesian and Frequentist methodology $\endgroup$
    – probabilityislogic
    Mar 25, 2011 at 22:51

5 Answers 5

Reset to default
13
$\begingroup$

First of all, you have to note the context: this only applies when the trial was stopped early due to interim monitoring showing efficacy/futility, not for some random outside reason. In that case the estimate of the effect size will be biased in a completely statististical sense. If you stopped for efficacy, the estimated effect will be too high (assuming it is positive), if you stopped for futility, it will be too low.

Piantodosi does give an intuitive explanantion as well (Sec 10.5.4 in my edition). Suppose the true difference in two means is 1 unit. When you run a lot of trials, and look at them at your interim analysis time, some of them will have observed effect sizes much above 1, some much below one, and most around 1 - the distribution will be wide, but symmetric. The estimated effect size at this point would not be very accurate, but would be unbiased. However you only stop and report an effect size if the difference is significant (adjusted for multiple testing), that is the estimate is on the high side. In all other cases you keep going and don't report an estimate. That means that conditional on having stopped early, the distribution of the effect size is not symmetric, and its expected value is above the true value of the estimate.

The fact that this effect is more severe early on comes from the larger hurdle for stopping the trial, thus a larger part of the distribution being thrown away during the conditioning.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ At first I thought this too, but when I sat down to prove it, I couldn't: I could only show that the resulting estimate actually is unbiased. (New intuition: the positive bias from a conditional stop balances a negative bias from carrying the experiment to completion.) So: can you present a more rigorous demonstration? $\endgroup$
    – whuber ♦
    Mar 25, 2011 at 14:26
  • $\begingroup$ @whuber I'll try to write it up, but the point is that Piantodosi's statement is only about what happens when you do stop early. There is not completion to balance it. $\endgroup$
    – Aniko
    Mar 25, 2011 at 14:56
  • 2
    $\begingroup$ @whuber Yes, that's what the original statement claims as well. Your point that there will be an opposite bias conditional on completing the study is also valid. The whole message should be that once you start doing interim monitoring, funny things start happening to your ability to estimate the effect size. $\endgroup$
    – Aniko
    Mar 25, 2011 at 15:07
  • 3
    $\begingroup$ @Aniko It should be possible to adjust for the bias when early termination occurs. We seem therefore to be discussing the naive use of a standard estimator, intended for fixed-size random samples, in conditionally terminated experiments, where such estimators do not have their desired properties. (+1, by the way.) $\endgroup$
    – whuber ♦
    Mar 25, 2011 at 15:17
  • 2
    $\begingroup$ @whuber Sure, you can adjust for this bias, but first you have to recognize that it exists. And then you have to sell to the investigator that even though clearly 5 out of 10 patients responded, the estimated rate of response is 40% (numbers made up) after adjusting for the bias due to early stopping. $\endgroup$
    – Aniko
    Mar 25, 2011 at 15:52
3
$\begingroup$

Here is an illustration of how bias might arise in conclusions, and why it may not be the full story. Suppose you have a sequential trial of a drug which is expected to have a positive (+1) effect but may have a negative effect (-1). Five guinea pigs are tested one after the other. The unknown probability of a positive outcome in a single case is in fact $\frac{3}{4}$ and a negative outcome $\frac{1}{4}$.

So after five trials the probabilities of the different outcomes are 

 Outcome     Probability
+5-0 = +5    243/1024
+4-1 = +3    405/1024
+3-2 = +1    270/1024
+2-3 = -1     90/1024
+1-4 = -3     15/1024
+0-5 = -5      1/1024

so the probability of a positive outcome overall is 918/1024 = 0.896, and the mean outcome is +2.5. Dividing by the 5 trials, this is an average of a +0.5 outcome per trial.

It is the unbiased figure, as it is also $+1\times\frac{3}{4}-1\times\frac{1}{4}$.

Suppose that in order to protect guinea pigs, the study will be terminated if at any stage the cumulative outcome is negative. Then the probabilities become 

 Outcome     Probability
+5-0 = +5    243/1024
+4-1 = +3    324/1024
+3-2 = +1    135/1024
+2-3 = -1     18/1024
+1-2 = -1     48/1024
+0-1 = -1    256/1024

so the probability of a positive outcome overall is 702/1024 = 0.6855, and the mean outcome is +1.953. If we looked the mean value of outcome per trial in the previous calculation, i.e. using $\frac{+5}{5}$, $\frac{+3}{5}$, $\frac{+1}{5}$, $\frac{-1}{5}$, $\frac{-1}{3}$ and $\frac{-1}{1}$ then we would get +0.184.

These are the senses in which there is bias by stopping early in the second scheme, and the bias is in the predicted direction. But it is not the full story.

Why do whuber and probabilityislogic think stopping early should produce unbiased results? We know the expected outcome of the trials in the second scheme is +1.953. The expected number of trials turns out to be 3.906. So dividing one by the other we get +0.5, exactly as before and what was described as unbiased.

Improve this answer
$\endgroup$
2
  • $\begingroup$ you are taking the perspective of "pre-data" world. What you say is true, that the stopping rule matters but only before you consider the data. This is because the stopping rule provides information about the data, but not about the true probabilities. So once the data is in, the stopping rule no longer matters. Note that the true probabilities are unknown in the actual experiment. So you also need to consider situations when the probabilities are, say $P(+)=\frac{1}{4}$ and $P(-)=\frac{3}{4}$, as well as any other possible combination. $\endgroup$
    – probabilityislogic
    Mar 25, 2011 at 21:59
  • $\begingroup$ So I take your example as stating that $P(H|S,I)\neq P(H|I)$. This is certainly true! My answer also conditions on $D$ though. This is because, if you tell me the stopping rule, but not whether you actually did stop, I can figure this out from the data set that I actually have. In fact, I can figure out if any stopping rule would have actually stopped, once I know the data. $\endgroup$
    – probabilityislogic
    Mar 25, 2011 at 22:07
1
$\begingroup$

Well, my knowledge on this comes from the Harveian oration in 2008 http://bookshop.rcplondon.ac.uk/details.aspx?e=262 Essentially, to the best of my recollection the results will be biased as 1) stopping early usually means that either the treatment was more or less effective than one hoped, and if this is positive, then you may be capitalising on chance. I believe that p values are calculated on the basis of the planned sample size (but i could be wrong on this), and also if you are constantly checking your results to see if any effects have been shown, you need to correct for multiple comparisons in order to insure that you are not merely finding a chance effect. For example, if you check 20 times for p values below .05 then statistically speaking, you are almost certain to find one significant result.

Improve this answer
$\endgroup$
2
  • $\begingroup$ PART 1 First of all, thank you for your answer. Indeed, frequentist methods correct for multiple testing. Hence, the problem of biased treatment effect estimate cannot come from there. At an interim analysis, the test is based on the current information, using the current sample size, not the overall planned sample size. So the problem does not come from there either. $\endgroup$
    – ocram
    Mar 25, 2011 at 10:28
  • $\begingroup$ PART2 I agree that stopping early may mean that the treatment is "more effective than one hopped". In that sense, the estimate treatment effect would be larger than expected. But, according to me, this does not make it biased... Instead, according to me, in some sense, "our hope was biased". $\endgroup$
    – ocram
    Mar 25, 2011 at 10:30
1
$\begingroup$

I would disagree with that claim, unless by "bias" Piantadosi means that part of the accuracy which is commonly called bias. The inference won't be "biased" because you chose to stop per se: it will be "biased" because you have less data. The so called "likelihood principle" states that inference should only depend on data that was observed, and not on data that might have been observed, but was not. The LP says

$$P(H|D,S,I)=P(H|D,I)$$

Where $H$ stands for the hypothesis you are testing (in the form of a proposition, such as "the treatment was effective"), $D$ stands for the data you actually observed, and $S$ stands for the proposition "the experiment was stopped early", and $I$ stands for the prior information (such as a model). Now suppose your stopping rule depends on the data $D$ and on the prior information $I$, so you can write $S=g(D,I)$. Now an elementary rule of logic is $AA=A$ - saying that A is true twice is the same thing as saying it once. This means that because $S=g(D,I)$ will be true whenever $D$ and $I$ are also true. So in "boolean algebra" we have $D,S,I = D,g(D,I),I = D,I$. This proves the above equation of the likelihood principle. It is only if your stopping rule depends on something other than the data $D$ or the prior information $I$ that it matters.

Improve this answer
$\endgroup$
8
  • $\begingroup$ @probabilityislogic: Thank you! If I understand it well, "bias" should not be taken in a statistical sense. I think this makes sense because Piantadosi speaks about the "bias" of an estimate and not of an estimator... $\endgroup$
    – ocram
    Mar 25, 2011 at 13:10
  • $\begingroup$ @ocram - What I meant by "biased" is the usual statistical term $E(\mu-\hat{\mu})^{2}=var(\hat{\mu})+Bias(\hat{\mu})$ where $\mu$ is the "true value" and $\hat{\mu}$ is the "estimator". If the second term (the bias) depends on the sample size, then you would expect that stopping early would increase the bias, because it has decreased the sample size, relative to if the experiment continued. But from what you say, it sounds like "bias" should be interpreted as "error" from Piantadosi's perspective. $\endgroup$
    – probabilityislogic
    Mar 25, 2011 at 14:01
  • 1
    $\begingroup$ This argument says nothing about the bias, only the hypothesis testing aspect of the problem, which nobody questions. $\endgroup$
    – Aniko
    Mar 25, 2011 at 14:12
  • $\begingroup$ @Prob I have to agree with @Aniko: it is obvious that when the null is true, there is a positive probability of early termination, in which case the estimate of the effect will be nonzero. Thus the expectation of the estimated effect, conditional on early termination, is positive, whereas the unconditional expectation is zero. (Notice that the OP is addressing estimation, not hypothesis testing.) $\endgroup$
    – whuber ♦
    Mar 25, 2011 at 15:02
  • $\begingroup$ @whuber - there is no difference between estimation and hypothesis testing here, just replace $H$ by the proposition "the true value of $\mu$ is in some small interval $(a,a+da)$". The estimate depends on $S$ only through the data $D$ and prior $I$. So while this may be true before you see the data (that $S$ matters), $S$ is irrelevant after the data has been observed. $S$ only gives you information about $D$ and $I$, not directly about $\mu$. $\endgroup$
    – probabilityislogic
    Mar 25, 2011 at 15:41
0
$\begingroup$

there will be bias (in "statistical sense") if termination of studies is not random.

In a set of experiments run to conclusion, the "early on" results of (a) some experiments that ultimately find "no effect" will show some effect (as a result of chance) and (b) some experiments that ultimately do find an effect will show "no effect" (likely as a result of lack of power). In a world in which you terminate trials, if you stop (a) more often than (b), you'll end up across run of studies with bias in favor of finding an effect. (Same logic applies for effect sizes; terminating studies that show "bigger than expected" effect early on more often than ones that show "as expected or lower" will inflate count of findings of "big effect.")

If in fact medical trials are terminated when early results show a positive effect -- in order to make treatment available to subjects in placebo or others -- but not when early results are inconclusive, then there will be more type 1 error in such testing than there would be if all experiments were run to conclusion. But that doesn't meant the practice is wrong; the cost of type 1 error, morally speaking, might be lower than denying treatment as quickly as one otherwise would for treatments that really would be shown to work at end of full trial.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Please see my comment to Aniko's reply, because I would ask the same question of you: can you provide a more rigorous demonstration? $\endgroup$
    – whuber ♦
    Mar 25, 2011 at 14:27
  • $\begingroup$ I defer to Aniko--he does a better job than I could. But if you agree that "desk drawer effect" results in bias, the logic here is identical. There is bias in favor of data supportive of hypothesis -- in the former case b/c the not-supportive data are not reported, in the latter b/c some fraction of not-supported data is necessarily not being collected: Ending the trial early when results look good excludes that part of the "bad results" distribution populated by trials that will produce their bad results late. Maybe this bias can be adjusted for--but there is bias in need of adjustment. $\endgroup$
    – dmk38
    Mar 25, 2011 at 23:29
  • $\begingroup$ @dmk I'm just trying to spur you both to have a debate with @Probability, with whom you seem to sharply disagree ;-). $\endgroup$
    – whuber ♦
    Mar 26, 2011 at 1:44
  • 1
    $\begingroup$ @whuber, @dmk - I think we are destined to disagree not because either of us is wrong, but because each is answering a different question. The frequentist considers $P(D|H,S,I)$ as "the answer", and if this is the object, then the stopping rule does matter. But to what question is it the answer? To me, this answers the question: "what data are we likely to observe, given the hypothesis is true (or parameter is said value), that we have stopped early, and from our prior information?" But this is not the question that is actually being asked I think (more later) $\endgroup$
    – probabilityislogic
    Mar 26, 2011 at 4:03
  • 1
    $\begingroup$ @probability That is one way to look at it. Another is to dodge the hypothesis altogether and address the question actually being asked; to wit, what is the size of the treatment effect? From this point of view termination can occur once the estimate is known with sufficient accuracy to support decision making. For example, we might want to have high confidence that the gain in health from prescribing the treatment is likely to exceed the costs (and side effects) of the treatment. $\endgroup$
    – whuber ♦
    Mar 26, 2011 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.