8
$\begingroup$

It is not uncommon for biologists to repeat experiments to confirm initial results. Intuitively this makes some sense but to me seems inefficient and potentially problematic.

(For example, one client brought data from 'inconsistent' experiments that 'worked' 50% of the time and didn't work 50% of the time. They were looking for differences in the experimental conditions but it turns out the power of their set-up was about 50%, there was no inconsistency in the effect sizes and they could have easily stopped after the first few runs.)

If you design an experiment with a power of 80% at p<0.05 but require it to succeed three times out of three before you can claim a finding then

  1. the chance of 'inconsistent' results (in terms of statistical significance at least) is high
  2. your effective power for the whole procedure is $0.8^3=0.51$
  3. the size is much smaller (about 0.0001) than the nominal 5% rate
  4. reporting becomes more difficult

So, I have usually recommended that instead of planning for repeated experiments, researchers use the available resource to properly design single larger experiments, using blocks where they might have used experimental repeats. In a way the set-up is very similar but the analysis and reporting is different.

But I saw today that the guidance from the UK Medical Research Council on funding applications for animal experiments does contain an explicit reference to repeating experiments to protect against spurious false positives. That is (from https://www.ukri.org/councils/mrc/guidance-for-applicants/proposals-involving-animal-use/4-3-experimental-design-avoidance-of-bias-and-statistical-considerations/#contents-list):

[we require] an indication of the number of independent replications of each experiment to be performed with the objective of minimising the likelihood of spurious non replicable results. If there are no plans for studies to be independently replicated within the current proposal then this will need to be justified.

There's also generic guidance like this one: https://www.sciencebuddies.org/science-fair-projects/competitions/experimental-design-increasing-signal-to-noise, and I've seen at least one design textbook allude to the fact that repeating experiments increases confidence in findings, without examining this formally.

Note this isn't repeating across contexts to increase generalisability, it seems to be a suggestion or requirement that a lab simply performs an identical experiment a number of times.

So I wonder if I'm missing something or misunderstanding the guidance. If you want a false positive rate below 5% then why not just design well and set alpha accordingly? Is this just to guard against spurious results from poor design? But if there's something wrong with the design of experiment 1 then why would experiments 2 and 3 help (if they're designed identically).

Why might it be better to plan to repeat experiments within your lab (each with its own analysis) instead of designing and analysing a single larger experiment properly? And if repeating experiments is a preferable approach, how would you design that 'meta-experiment' (number and nature of repeats) and how would you report it?

Edit: This short paper (Cummins 1999) discusses the issue but frames it as "report your repeats as blocks" rather than "use blocking instead of planning repeats". https://www.cell.com/trends/plant-science/fulltext/S1360-1385(99)01439-9.

It may be then that to meet the guidance we discuss doing repeats, but treating them as blocks and providing a single power calculation for all 'repeats' combined.

I am still very open to answers on any aspect of this issue, in particular when it might make sense to recommend a lab 'repeats' experiments as opposed to designing single larger blocked experiments instead, or whether the MRC guidance should be amended.

$\endgroup$
6
  • 4
    $\begingroup$ In the UK MRC quote, are you sure that "independent replications of each experiment" means new replications by the same lab? I usually think of this in terms of collaborating across several labs. If different people in different buildings get similar results, it's probably not just a fluke of one lab's room setup or personnel or whatever. That's a different issue than the sampling-error concerns you raise in terms of power and level (which are also important, just different!) $\endgroup$
    – civilstat
    Dec 14, 2023 at 14:17
  • 3
    $\begingroup$ (For example, I've heard of a fish behavior study that wasn't replicating well. It turned out that one lab tech was very short and the fish didn't see her when she walked over to observe them, but the other lab's staff were taller and the fish did see them, and that was enough to change whatever behavior was being studied.) $\endgroup$
    – civilstat
    Dec 14, 2023 at 14:20
  • 1
    $\begingroup$ @civilstat You could be right, but most applications wouldn't be recruiting extra partners to run the same experiments in different labs in the context of a single funding application (as far as I know). People tend to recruit more labs when they need an extra capability. If they were I'd probably still be inclined to consider this as one multi-site study rather than independent replications! $\endgroup$ Dec 14, 2023 at 14:20
  • 4
    $\begingroup$ Treating it as one multi-site study and blocking on site sounds like a sensible approach to me! $\endgroup$
    – civilstat
    Dec 14, 2023 at 14:21
  • 1
    $\begingroup$ @kjetilbhalvorsen Thank you, but I like EdM's answer better than my own comments! $\endgroup$
    – civilstat
    Mar 5 at 2:55

1 Answer 1

4
$\begingroup$

Why might it be better to plan to repeat experiments within your lab (each with its own analysis) instead of designing and analysing a single larger experiment properly? And if repeating experiments is a preferable approach, how would you design that 'meta-experiment' (number and nature of repeats) and how would you report it?

There are several reason for doing repeat biological experiments within a single lab.

First, there are almost always uncontrolled unknown factors in biological experiments that can vary among repeats. (I've spent the past 5 decades being fooled in many ways by Mother Nature.) You might set up a massive multi-block design to be performed all at once, not knowing that the results depend on one particular uncontrolled factor. Then even you, let alone another lab, might not be able to repeat the results later. The separate repeats at least provide a bit of assurance that some otherwise uncontrolled factors are being averaged over.

Second, many biology labs have only a handful of staff. Graduate students often have to do all the experiments by themselves. It can be impossible in practice to carry out the type of massive multi-block design that you have in mind. In those circumstances, doing simpler experiments over a course of days to weeks is more practical.

Third, given the problems with reproducibility of life-science studies, I'm not sure that the extra stringency of requiring a "significant" result multiple times in the same lab is a bad thing.

The design of a multi-repeat study depends on the type of experiment and the expectations of the specific scientific peer group. The design and expectations might differ, say, between animal experiments and cell-culture studies.

The way to report repeated experiments also depends on the nature of the study. Some types of study might well be handled by treating the repeats as blocks, as suggested in the question. Others are best documented by illustrating details of a single representative experiment in the main text, specifying the total number of repeats, and perhaps showing the other repeats in supplementary material.

$\endgroup$
2
  • $\begingroup$ Thanks for the considered answer. I agree these would be the main arguments in favour of repeating experiments. But I'm not sure any of these scenarios addresses the question of why a single blocked experiment wouldn't be a better solution to the issue. Given the 'uncontrollable factors' in the first case I would still prefer to block over these factors and average over them. There's no reason blocks would have to be run simultaneously (in fact it would be unusual I think). In the second scenario with limited staff, again there's no reason not to spread out blocks between people. $\endgroup$ Mar 5 at 8:42
  • $\begingroup$ In any case, I don't think we're argument so much about design as the interpretation of the subsequent data, that is whether it's considered all together or in individual pieces. But thanks again for the answer. $\endgroup$ Mar 5 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.