6
$\begingroup$

I have read that in group sequential designs, having a maximum of $k$ stops, for the $j$-th stop $j < k$ there are 4 bounds chosen $a_j,b_j, c_j, d_j$ such that the three decisions are considered. A set of lecture notes supporting this concept can be found here.

  1. Reject $H_=, H_+$ in favor of $H_-$ if $T<a_j$ (reject futility, benefit in favor of harm).
  2. Reject $H_-, H_+$ in favor of $H_=$ if $b_j < T < c_j$ (reject harm, benefit in favor of futility).
  3. Reject $H_-, H_=$ in favor of $H_+$ if $T > d_j$ (reject harm, futility in favor of benefit).

Otherwise continue until the next stop.

What's the point of having a futility hypothesis? It seems to me that it's either a negative study outcome or a positive study outcome: thus, it should be specified as such a priori. If futility is "negative" it means the sponsor has no interest in moving on to market their drug because it's no better than existing therapies. If that's the case, then $T > a_j$ but $T< b_j$ seems like rationale to stop the study, because I don't care too much whether there's harm or futility: I'm not going to sell this drug.

If futility is "positive" it means the sponsor is interested in shifting their design strategy toward proving non-inferiority since their new drug is perhaps cheaper or better tolerated or has fewer interactions. In that case, it seems to me that they did the wrong type of study. They should have conducted a non-inferiority design to begin with.

$\endgroup$
2
  • 1
    $\begingroup$ AdamO where are you reading about the language "futility" hypothesis, I haven't encountered that one yet. :) $\endgroup$
    – Alexis
    Commented May 16, 2018 at 16:12
  • 1
    $\begingroup$ @Alexis I edited the question to include a link to a set of lecture notes here. $\endgroup$
    – AdamO
    Commented May 16, 2018 at 16:25

1 Answer 1

3
+100
$\begingroup$

What's the point of having a futility hypothesis?

Government regulations and medical ethics affect the design of medical studies and govern the conduct of clinical trials by describing good clinical practices (GCPs) for studies with both human and non-human animal subjects.

Examples and supporting quotes from the articles:

"Method

Conditional power is used to assess futility or the inability of the trial to show non-inferiority at the final analysis, by calculating the probability of demonstrating non-inferiority in the final analysis conditional on the observed results and upon assumptions on the future results of the trial. The Bullous Pemphigoid Steroids and Tetracyclines Study (BLISTER) trial is an example of a trial where a simultaneous inferior and non-inferior result could occur. A method for calculating conditional power for non-inferiority using simulations is described and applied at a hypothetical interim analysis of this trial.

Results

Stopping guidelines for futility based on conditional power are shown to be better suited to non-inferiority trials than the typical methods used in superiority trials. Simulations are a straightforward and flexible way of calculating conditional power.

Conclusions

Conditional power is an appropriate tool for defining stopping guidelines for futility in non-inferiority trials, particularly those with large non-inferiority margins.".

"Abstract

Noninferiority trials are intended to show that the effect of a new treatment is not worse than that of an active control by more than a specified margin. These trials have a number of inherent weaknesses that superiority trials do not: no internal demonstration of assay sensitivity, no single conservative analysis approach, lack of protection from bias by blinding, and difficulty in specifying the noninferiority margin. Noninferiority trials may sometimes be necessary when a placebo group can not be ethically included, but it should be recognized that the results of such trials are not as credible as those from a superiority trial.

Terminology

'Noninferiority' is a relatively new term that has not been universally adopted, and in the past noninferiority and equivalence trials, which have an important distinction, have both been referred to as 'equivalence trials'. To make the confusion even worse, both of these terms are somewhat misleading.

It is fundamentally impossible to prove that two treatments have exactly equivalent effects. Equivalence trials, therefore, aim to show that the effects differ by no more than a specific amount. This tolerance is known as the equivalence margin, and is often denoted by the symbol δ. In an equivalence trial, if the effects of the two treatments differ by more than the equivalence margin in either direction, then equivalence does not hold. Noninferiority trials, on the other hand, aim to show that an experimental treatment is not worse than an active control by more than the equivalence margin. An improvement of any size fits within the definition of noninferiority. Bioequivalence trials are true equivalence trials, but it is difficult to imagine any trial comparing the clinical effects of an experimental treatment and active control that would not more appropriately be termed a noninferiority trial.".

"Background

Many clinical trials show no overall benefit. We examined futility analyses applied to trials with different effect sizes.

Methods

Ten randomised cancer trials were retrospectively analysed; target sample size reached in all. The hazard ratio indicated no overall benefit (n=5), or moderate (n=4) or large (n=1) treatment effects. Futility analyses were applied after 25, 50 and 75% of events were observed, or patients were recruited. Outcomes were conditional power (CP), and time and cost savings.

Results

Futility analyses could stop some trials with no benefit, but not all. After observing 50% of the target number of events, 3 out of 5 trials with no benefit could be stopped early (low CP ≤ 15%). Trial duration for two studies could be reduced by 4-24 months, saving £44 000-231 000, but the third had already stopped recruiting, hence no savings were made. However, of concern was that 2 of the 4 trials with moderate treatment effects could be stopped early at some point, although they eventually showed worthwhile benefits.

Conclusions

Careful application of futility can lead to future patients in a trial not being given an ineffective treatment, and should therefore be used more often. A secondary consideration is that it could shorten trial duration and reduce costs. However, studies with modest treatment effects could be inappropriately stopped early. Unless there is very good evidence for futility, it is often best to continue to the planned end.".

"Abstract

Noninferiority trials test whether a new experimental treatment is not unacceptably less efficacious than an active control treatment already in use. With continuous improvements in health technologies, standard care, and clinical outcomes, the incremental benefits of newly developed treatments may be only marginal over existing treatments. Sometimes assigning patients to a placebo is unethical. In such circumstances, there has been increasing emphasis on the use of noninferiority trial designs. Noninferiority trials are more complex to design, conduct, and interpret than typical superiority trials. This paper reviews the concept of noninferiority trials and discusses some important issues related to them.".

Please continue reading at the links provided.

"... I don't care too much whether there's harm or futility: I'm not going to sell this drug."

There's a big difference between harm and futility, the placebo effect is a necessary consideration in some types of trials. These tests are done long before sales are approved and in some cases drugs that are harmful in certain circumstances are life saving in others. That is why some designs call for three categories, later in the trials only effective drugs are used as harmful or placebo products are already removed from the trial.


Use of conditional power alone to guide stopping would probably be incorrect. Years of education and study of trial methods are necessary to design competent clinical trials. There are too many things to consider and guidance by a team of experts is necessary, reliance on a Stack Exchange answer is ill advised. Indeed neither this site nor this author offers medical advice, this is mearly an attempt to provide a brief answer to laypersons whom share interest in the question and seek guidance for months of further reading.

For a discussion of other statistical methods see the following paper where the authors explore settings in which Bayesian predictive probabilities are advantageous for interim monitoring compared to Bayesian posterior probabilities, p-values, conditional power, or group sequential methods.

"The utility of Bayesian predictive probabilities for interim monitoring of clinical trials" (May 28 2014) by Benjamin R. Saville, Jason T. Connor, Gregory D Ayers, and JoAnn Alvarez. DOI: 10.1177/1740774514531352

Background

Bayesian predictive probabilities can be used for interim monitoring of clinical trials to estimate the probability of observing a statistically significant treatment effect if the trial were to continue to its predefined maximum sample size.

...

Conclusions

The use of Bayesian predictive probabilities enables the choice of logical interim stopping rules that closely align with the clinical decision making process.

...

Two common types of questions addressed by interim analyses include 1) Is there convincing evidence in favor of the null or alternative hypotheses? and 2) Is the trial likely to show convincing evidence in favor of the alternative hypothesis if additional data are collected? The first question pertains to the evidence presently shown by the data, and is best addressed using estimation, p-values, or Bayesian posterior probabilities. The second deals with prediction of what evidence will be available at later stages in the study, and is best addressed using stochastic curtailment methods such as conditional power or predictive probability.

Conditional power is often criticized for assuming the unknown parameters are fixed at specific values. In contrast, Bayesian predictive probabilities average the probability of trial success over the variability in the parameter estimates, and can be used whether the trial’s primary analysis is frequentist or Bayesian. They are most often used to predict trial success at a final pre-determined sample size, but can also be used to predict trial success based on an interim sample size with extended follow-up. Predictive probabilities have been widely discussed in the literature. However, the literature lacks a general discussion of the advantages of predictive probabilities for interim monitoring. This article contrasts predictive probabilities versus traditional methods for interim monitoring of clinical trials. We illustrate settings in which predictive probabilities have advantages over more traditional methods of interim monitoring, particularly in the context of futility monitoring and efficacy monitoring with lagged outcomes (Sections 2-4). We also explore the relationship between the predictive probability and posterior probability (Section 5) and conclude with a discussion.".

$\endgroup$
2
  • $\begingroup$ This is very good! Let me summarize: 1. Equivalence/non-inferiority is really a different branch/form of research w/ different rationale and conduct (perhaps more stringent safety monitoring) 2. conditional power is the basis of the stopping rule, if it's too low we stop the trial because why waste time and harm subjects with exposures? 3. We don't really conclude that the treatments are the same, but at the very least we raised no flag that the IND is harmful, and so further research can be done. $\endgroup$
    – AdamO
    Commented May 17, 2018 at 11:35
  • $\begingroup$ @AdamO - I've updated the answer with additional information about stopping and a bit of info about how such decisions might be guided in some cases. Back for another edit. I will attempt to see if there is some means to condense the answer without leaving out important points nor reducing it to a long list of links with little textual answer. $\endgroup$
    – Rob
    Commented May 18, 2018 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.