Is there a good reason for a lab to repeat experiments instead of conducting a single larger blocked experiment

Question

It is not uncommon for biologists to repeat experiments to confirm initial results. Intuitively this makes some sense but to me seems inefficient and potentially problematic.

(For example, one client brought data from 'inconsistent' experiments that 'worked' 50% of the time and didn't work 50% of the time. They were looking for differences in the experimental conditions but it turns out the power of their set-up was about 50%, there was no inconsistency in the effect sizes and they could have easily stopped after the first few runs.)

If you design an experiment with a power of 80% at p<0.05 but require it to succeed three times out of three before you can claim a finding then

1. the chance of 'inconsistent' results (in terms of statistical significance at least) is high
2. your effective power for the whole procedure is $0.8^3=0.51$
3. the size is much smaller (about 0.0001) than the nominal 5% rate
4. reporting becomes more difficult

So, I have usually recommended that instead of planning for repeated experiments, researchers use the available resource to properly design single larger experiments, using blocks where they might have used experimental repeats. In a way the set-up is very similar but the analysis and reporting is different.

But I saw today that the guidance from the UK Medical Research Council on funding applications for animal experiments does contain an explicit reference to repeating experiments to protect against spurious false positives. That is (from https://www.ukri.org/councils/mrc/guidance-for-applicants/proposals-involving-animal-use/4-3-experimental-design-avoidance-of-bias-and-statistical-considerations/#contents-list):

[we require] an indication of the number of independent replications of each experiment to be performed with the objective of minimising the likelihood of spurious non replicable results. If there are no plans for studies to be independently replicated within the current proposal then this will need to be justified.

There's also generic guidance like this one: https://www.sciencebuddies.org/science-fair-projects/competitions/experimental-design-increasing-signal-to-noise, and I've seen at least one design textbook allude to the fact that repeating experiments increases confidence in findings, without examining this formally.

Note this isn't repeating across contexts to increase generalisability, it seems to be a suggestion or requirement that a lab simply performs an identical experiment a number of times.

So I wonder if I'm missing something or misunderstanding the guidance. If you want a false positive rate below 5% then why not just design well and set alpha accordingly? Is this just to guard against spurious results from poor design? But if there's something wrong with the design of experiment 1 then why would experiments 2 and 3 help (if they're designed identically).

Why might it be better to plan to repeat experiments within your lab (each with its own analysis) instead of designing and analysing a single larger experiment properly? And if repeating experiments is a preferable approach, how would you design that 'meta-experiment' (number and nature of repeats) and how would you report it?

Edit: This short paper (Cummins 1999) discusses the issue but frames it as "report your repeats as blocks" rather than "use blocking instead of planning repeats". https://www.cell.com/trends/plant-science/fulltext/S1360-1385(99)01439-9.

It may be then that to meet the guidance we discuss doing repeats, but treating them as blocks and providing a single power calculation for all 'repeats' combined.

I am still very open to answers on any aspect of this issue, in particular when it might make sense to recommend a lab 'repeats' experiments as opposed to designing single larger blocked experiments instead, or whether the MRC guidance should be amended.

Answer 1

Why might it be better to plan to repeat experiments within your lab (each with its own analysis) instead of designing and analysing a single larger experiment properly? And if repeating experiments is a preferable approach, how would you design that 'meta-experiment' (number and nature of repeats) and how would you report it?

There are several reason for doing repeat biological experiments within a single lab.

First, there are almost always uncontrolled unknown factors in biological experiments that can vary among repeats. (I've spent the past 5 decades being fooled in many ways by Mother Nature.) You might set up a massive multi-block design to be performed all at once, not knowing that the results depend on one particular uncontrolled factor. Then even you, let alone another lab, might not be able to repeat the results later. The separate repeats at least provide a bit of assurance that some otherwise uncontrolled factors are being averaged over.

Second, many biology labs have only a handful of staff. Graduate students often have to do all the experiments by themselves. It can be impossible in practice to carry out the type of massive multi-block design that you have in mind. In those circumstances, doing simpler experiments over a course of days to weeks is more practical.

Third, given the problems with reproducibility of life-science studies, I'm not sure that the extra stringency of requiring a "significant" result multiple times in the same lab is a bad thing.

The design of a multi-repeat study depends on the type of experiment and the expectations of the specific scientific peer group. The design and expectations might differ, say, between animal experiments and cell-culture studies.

The way to report repeated experiments also depends on the nature of the study. Some types of study might well be handled by treating the repeats as blocks, as suggested in the question. Others are best documented by illustrating details of a single representative experiment in the main text, specifying the total number of repeats, and perhaps showing the other repeats in supplementary material.