As per the title, I would like to know what are currently the major issues and controversies in clinical trials design and analysis.

  • $\begingroup$ Community wiki? $\endgroup$
    – Dave
    Nov 9, 2023 at 15:54

2 Answers 2


There's a lot of them and some might be the same ones people would have mentioned 10 or 20 years ago, but here's a few:

  • Choice of effect measure: some effect measures have good properties for estimating causal effects even under model misspecification, but may not apply to any one patient and may not transfer to patients at different levels at risk/different populations with different risk profiles (they may also be a lot less statistically efficient)
    • Examples: Risk difference for a binary outcome vs. odds ratio, risk difference at 1 year for survival data vs. hazard ratio for full follow-up etc.
  • Population average vs. individual effects: Are we more interested in what happens to individuals vs. a population average (may be the same for some models, but not for others)? Does the answer perhaps depend on the stakeholder asking the question? For population average effects, should they be for the trial population or some other population? Presumably a physician / patient is more interested in individual effects, while insurance companies or national health systems are more interested in population average (but perhaps for their particular population, rather than the population enrolled in a trial).
  • Choice of estimand: Under what circumstances is a treatment policy vs. hypothetical estimand sensible/of interest? Do prescribers/patients/insurers want to know what happens when people continue taking a drug or averaged across a population part of which has stopped taking a drug (because some people always do)? For answering this, does it matter why people stopped (e.g. 1) because of side effects = likely would also stop in clinical practice, 2) because they were tired of the trial procedures, but it had nothing to do with the drug under study = might not stop in clinical practice, 3) what if we don't know?)
  • Choice of design/data source:
    • How much do we want to rely on often non-randomized real world data vs. randomized trials (which could sometimes of course also be made more representative of the real world in various respects, but sometimes might already be reasonably representative)? How do we trade-off representativeness of the population vs. internal validity of the study? To what extent is it a real concern that treatment effects compared with control would truly change in different populations vs. to what extent is the potential for bias from observational data a much bigger concern (probably pretty often?)?
    • Can we use external control groups (including of course in the scenario with some internal randomized control), what are the best sources (other trials, real-world data sources) and what should want do to account for differences (e.g. matching on some things, which could include longitudinal history information)?
  • machine learning/"AI": What can these contribute and where would we not want to rely on them? E.g.
    • Patient identification (inclusion/exclusion) for trials and would such inclusion criteria then be reflected in labeling? Would it be appropriate to withhold drugs from those not identified by such a model?
    • Outcome prediction so that we adjust for it in analysis?
    • Somehow making trial analysis more efficient?
    • Finding subgroups where drugs were better/worse (the challenge is the noisiness of the process, which makes findings very shaky/hard to be sure about - can you somehow make this more stable based on incorporating prior knowledge about the world/biology/medicine/...?)
    • Even if we just talk about prediction purposes, under which circumstances do more complex model reliably and predictably beat simple linear/logistic/etc. regression models by a meaningful margin that justifies their use?
    • Can you automate endpoint assessment? E.g. assessing cancer disease progression from imaging, inferring outcomes from wearable devices etc.?
    • Many more ideas, some of which are completely misguided, some of which might be good.
  • Model informed drug development: How far should you go? Could you (and do you want to) do the confirmatory analysis of a registration study using a pre-specified PK/PD model? If you can't fully pre-specify it, how much of a problem is it? Exactly when can you avoid running a dose finding study by PK/PD modeling (possibly incl. based on non-human data)?
  • How to size dose finding studies: What is a good operating characteristic of study designs to consider when looking at how large a dose finding study should be? The power to show a difference of drug vs. control is probably not it, but what is it?
  • Platform studies: When are which of these (multiple drugs one disease, multiple diseases one drug, multiple drugs in multiple diseases, these options but with disease sub-types...) useful/preferable to traditional development options?
  • Data sharing: How to make it easy, ethically appropriate, give due creditors to original researchers, how to prevent misuse? How should it be handled that people state that "data are available upon request", but they never respond to requests? How to best ensure patients' consent/that things don't happen inappropriately?
  • Use of published information: How can one use all sorts of information that get published, but are not original individual patient data? Examples: aggregate summary statistics, figures of information, statements in press releases that are limited to statistical significance and so on.
  • Appropriate summaries of safety information (such as occurrence of adverse events), where most people would agree simple percentages don't tell the full story, but what to do instead is under debate.

Randomisation: Proper randomisation is crucial for preventing selection bias and ensuring that the treatment and control groups are comparable. Inadequate randomisation can lead to skewed results and potentially erroneous conclusions. This should really not be a problem, and yet studies do get retracted for issues with randomisation. For example see here.

Sample Size: Determining the appropriate sample size is essential for ensuring that a study has enough power to detect a true effect. Too small a sample size can result in a study being underpowered and failing to detect significant differences, while too large a sample can be wasteful and unethical. This can be problematic when different sample size software gives quite different numbers for sample size. For example see here.

Blinding: Blinding (or masking) is the process of keeping trial participants, and sometimes the researchers, unaware of which treatment each participant is receiving to prevent bias. Inadequate blinding can introduce bias and affect the results of the trial. Sometimes, it may be impossible to achive proper blinding, for example in trials comparing surgical vs non-surgical interventions, different types of talking therapies, or where a treatment has distinctive side-effects.

Data Dredging: This occurs when researchers sift through data looking for statistically significant patterns, which may result in spurious findings. It's also known as p-hacking or fishing expeditions.

Missing Data: Participants often drop out of trials or miss scheduled assessments, which can result in missing data. How this missing data is handled can significantly impact the results and interpretations of the trial. Techniques such as multiple imputation can help to mitigate this issue, but that in itself has various issues in implementation, and may still result in bias.

Multiplicity: Also known as the multiple comparisons problem, this arises when researchers test multiple hypotheses or conduct multiple sub-group analyses without proper statistical adjustments, increasing the risk of false-positive findings.

Publication Bias: Positive or significant results are more likely to be published than negative or non-significant ones, leading to a biased representation of evidence in the literature.

Selective Reporting: Sometimes, researchers may report only certain outcomes from a study, omitting others that are less favourable or not statistically significant.

Conflicts of Interest: Financial or other interests can influence how a study is designed, conducted, or reported. Transparency about potential conflicts of interest is critical to the integrity of clinical research.

Ethical Considerations: There are often ethical concerns about placebo use (especially when effective treatments exist), informed consent, and the balance of risks and benefits for participants.

External Validity: The ability to generalise the results of a trial to other populations (external validity) can be limited if the study population is not representative of the broader patient population who would use the intervention.

Statistical Significance vs. Clinical Significance: There is often a debate about the interpretation of statistical significance in the absence of clear clinical relevance. A result can be statistically significant without being clinically meaningful.

Adaptive Designs: While adaptive designs can make clinical trials more flexible by allowing for modifications to the trial after it has started based on interim results, they can also introduce complexity and potential bias.

Patient-Reported Outcomes: The subjective nature of patient-reported outcomes can introduce variability and affect the reliability of the data.

Transparency and Reproducibility: The reproducibility crisis in science, where other researchers cannot reproduce the results of a study, affects clinical trials as well. There is a push for more transparency in methods and data to ensure that results can be independently verified.

Many of these issues have been known about for many years, and reflect the ongoing challenges in designing and implementing clinical trials that are robust, reliable, and ethically sound. Addressing these concerns often requires careful planning, clear protocols, and rigorous statistical analysis, as well as adherence to ethical standards and guidelines. The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH), publishes global standards for the design, conduct, safety, and reporting of clinical trials to ensure that pharmaceuticals are effective and safe, facilitating international regulatory approval, and following those publications will help to mitigate most if not all of the above.


Not the answer you're looking for? Browse other questions tagged or ask your own question.