3
$\begingroup$

I'm hoping to hear from someone who has worked on mouse models or similar biological analyses where there is a tendency to run 'replicates' of an experiment. I know multiple testing is a sizeable kettle of fish which is definitely relevant to this discussion. I have some applications for projects where they talk about running 3 replicates of an experiment, where each experiment has n = 3 to 7. However there seems to be no mention of what they will do with the multiple sets of results, as in how they will handle success, failure, success vs failure, failure, success etc. It seems like this 'replicates' approach is quite common practise in this field.

What are your thoughts / experiences with this situation?

I know there are different types of replication, technical vs biological, however I've found little useful reading on this issue.

$\endgroup$
2
$\begingroup$

First of all, I'm not sure whether the replicates are really what a statistician would call replicates (statisticians please comment). Mice that are grown together (same supplier, same cage, or even same litter) are often more similar to each other than mice from different cages, suppliers, litters. So the "cage" is a confounding variable that you otherwise neglect to measure. In consequence, I woudn't say that you have 3 separate result sets and would go for Prof. Harrell's grand model.

Obviously whether you want to experiment 3 x on 5 mice or 5 x on 3 mice depends on the between cage and between mice variance (plus the fact that 3 x 5 is cheaper than 5 x 3).

$\endgroup$
1
$\begingroup$

The first thing that comes to my mind when I read of the approach that you describe is that there is a miss-match between the idea of replicating an experiment and the use of "success" and "failure" as descriptors of the outcomes. Presumably a success would be a result that is significant in the Neyman-Pearson paradigm (i.e. P < alpha). However, that paradigm allows control of type I errors and specification of power by assuming that the experimenter will act as if the null hypothesis is false when a significant result is observed. In that case there is no point in testing it again--the determination of type I error rates assumes that you discard the hypothesis and move on. Neyman called the process 'inductive behavior'.

A consequence of inductive behavior is that you can't integrate the information across multiple replications of the overall experiment within the Neyman-Pearson paradigm. There would be no sensible interpretation of any sequence of significant and not significant results from repeats of the experiment using the Neyman-Pearson paradigm.

If you use instead Fisher's approach of inductive inference, then you can combine P values from multiple experiments testing the same null hypothesis and obtain a composite value. The P values are indices of the strength of evidence against the null hypothesis (and NOT error rates!) and so, just as it is sensible to combine the evidence from multiple sources before judgement, you should combine the P values. However, while there are several (many?) way to combine the P values, it is perhaps the case that none is perfect.

Arguably the best way to combine evidence from multiple experiments is to use a likelihood-based approach (see Royall 1997 for a very clear exposition) . The likelihood functions for each of the experiments can be combined exactly by multiplication to yield a composite function that can support both interval estimation and statement of the probability of the overall set of results having come from a true null hypothesis.

Royall, R. M. (1997). Statistical evidence: a likelihood paradigm. Chapman & Hall.


It is unfortunate that Fisher was frequently unclear in his expression, and rather inconsistent in his application of the ideas. However, he was undoubtedly in favor of repeating experiments and hated Neyman's idea of inductive behavior: “...we may be able to validly apply a test of significance to discredit a hypothesis the expectations from which are widely at variance with the ascertained fact. If we use the term rejection for our attitude to such a hypothesis, it would be clearly understood that no irreversible decision has been taken; that, as rational beings, we are prepared to be convinced by future evidence that appearances were deceptive, and that a remarkable and exceptional coincidence had taken place.” Stat Meth & Sci Inf p.37 (Chapter II section 4).

$\endgroup$
1
  • $\begingroup$ I would appreciate a discussion about why the 3 experiments would be analyzed separately, as opposed to building a grand model, and what is the best choice for the grand model. $\endgroup$ – Frank Harrell May 20 '11 at 13:48
1
$\begingroup$

In my case we ran an experiment 5 times in triplicates (to calculate SEM and mean) and selected the most attractive results

$\endgroup$
3
  • $\begingroup$ This response deserves some elaboration. What arguments can you add to motivate the use of a $5\times 3$ design, and why did you select the "most attractive results"? $\endgroup$ – chl Jun 20 '11 at 10:46
  • $\begingroup$ @chl: Why 5?. I agree the larger sample size we have the better. In my case We didn't make effect size planning and we considered the data obtained de factor violating the analysis de jure. Agree. And I can't argue with a professor for a long time yet:). And we saw the same effect in the 1st... 5th tests. And we decided to stop. Well, I always wanted to know why phrases like "the test was repeated twice with similar results", or "1 test representative of five is shown" even in the Nature journal. $\endgroup$ – starkid Jun 20 '11 at 11:16
  • $\begingroup$ @chl: (continued) If I'm not mistaken we need n-plicates, for example, in the case of RM ANOVA: when we have factors let say: "donor", "concentration", "substance" we can estimate not only a distinct factor but their combination (but I have a doubt...) $\endgroup$ – starkid Jun 20 '11 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.