4
$\begingroup$

This is somewhat of a regulatory question, rather than statistical question. But I assume that the readership of this community will be the most knowledgeable on this matter.

Question:

Is there any regulatory agency that actually documents submission of a statistical analysis plan (SAP) for clinical trials prior to the final clinical database lock and treatment unblinding?

What surprised me a bit is that there appears to be no obligation to actually submit a statistical analysis plan a priori. Could someone please tell me that I am incorrect about this and that a priori SAPs are always available to the public?

Here are some examples of rather vague suggestions:

ICH E9 Statistical Principles for Clinical Trials:

"The plan should be reviewed and possibly updated as a result of the blind review of the data and should be finalised before breaking the blind."

U.S. FDA, Guidance for Industry Developing Medical Imaging Drug and Biological Products, June 2004:

"We recommend that sponsors [...] submitted to the protocol before images have been collected."

Specific Background:

Recently, a "positive" phase III trial result has been published for a disease in my area of expertise. However, the are several "red flags" in my opinion:

  • Actual results are not reported
  • Only mixed model for repeated measures (MMRM) estimates were reported, but it is rather unclear what co-variates were used for "adjustment" in the model [on clinicaltrials.gov, there is no mentioning of "adjustment" for co-variates]
  • The outcome variable was transformed [on clinicaltrials.gov, there is mentioning of transformation of the outcome variable]

Having worked on natural history data for this disease before, it is obvious that one can easily find "significant" between-group differences by iterating across combinations of outcome transformations (no transformation, sqrt, log10) and co-variates combinations (age, gender, baseline lesion size). For obvious reasons, I suspected that a statistical analysis plan must be deposited somewhere a priori. However, the specific study did not publish the statistical analysis plan on clinicaltrials.gov.

$\endgroup$
2
  • 1
    $\begingroup$ It may make a difference whether you're asking about established tradition or recent practice. // Recently, IMHO in the US and elsewhere, a combination of factors has led to some departures from traditional standards of governmental oversight of clinical trials, and of premature announcements of claimed results that have not been peer reviewed. // As for Covid vaccines, I recently got my first shot and scheduled my 2nd anyhow. $\endgroup$
    – BruceET
    Feb 25 at 21:26
  • $\begingroup$ Thank you for the response. Would you have some more specific information regarding "legal documents"? (For the COVID trials BTW, information on the data and SAPs appears to be given [e.g., clinicaltrials.gov/ct2/show/NCT04368728], so this is not so much my concern.) $\endgroup$
    – William
    Feb 26 at 3:11
1
$\begingroup$

Focusing on Phase 3 studies only:

  1. SAPs are submitted to regulatory authorities along with the the protocol, informed consent documents, letters of intent, etc. etc. etc. for approval prior to the first patient initiated. The regulatory authorities may reject the submission if the SAP is not clearly aligned with the objectives.

  2. SAPs may be made publicly available after the database lock or publication of top-line results. There are many examples on clinicaltrials.gov.

  3. When submitting the CSR for regulatory approval by the respective agencies in accord with the ICH E9 guidance, the SAP is a controlled document meaning the cumulative revision history and the timeline are kept, and should match with the protocol. You cannot modify the SAP after database lock (OK you can but it's 99.9% likely to result in a rejection). If any aspect of the CSR is not in alignment with the SAP it is a major finding.

  4. It makes sense that the number of examples are relatively few. The authorities are hesitant to mandate that SAPs (or protocols even) are made publicly available. This is because the authorities are perceived as an immense hurdle in the drug approval process - a necessary one at that. I believe the authorities feel that if "successful" (meaning led to approval) study examples were too widely circulated, others would so closely copy the template they would fail to ever present or challenge new ideas, even when necessary. Indeed, you often hear that the FDA "won't approve a drug without a blinded controlled study" or "hates R and loves SAS" or "doesn't approve Bayesian trial designs but only frequentist". The opposite is true. But these new ideas need to be defensible to a number of highly experienced, cross-functional, and educated agents.

$\endgroup$
1
  • $\begingroup$ Thank you very much for this helpful response. Just to understand: If I still have the suspicion that an analysis was designed to benefit from vibration effects (through vagueness regarding variable transformations and co-variates) and authors responded to an email of me that the SAP can not be shared due to IP issues, what would be the next steps? (1.) forget about it, because U.S. FDA and EMA are well functioning, or (2.) letter to the editor regarding the journal article to highlight the vague aspects of the Methods section, (3.) something else $\endgroup$
    – William
    Mar 4 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.