Mouse models - 'replicates' and analysis I'm hoping to hear from someone who has worked on mouse models or similar biological analyses where there is a tendency to run 'replicates' of an experiment. I know multiple testing is a sizeable kettle of fish which is definitely relevant to this discussion. I have some applications for projects where they talk about running 3 replicates of an experiment, where each experiment has n = 3 to 7. However there seems to be no mention of what they will do with the multiple sets of results, as in how they will handle success, failure, success vs failure, failure, success etc. It seems like this 'replicates' approach is quite common practise in this field.
What are your thoughts / experiences with this situation?
I know there are different types of replication, technical vs biological, however I've found little useful reading on this issue.
 A: First of all, I'm not sure whether the replicates are really what a statistician would call replicates (statisticians please comment). Mice that are grown together (same supplier, same cage, or even same litter) are often more similar to each other than mice from different cages, suppliers, litters. So the "cage" is a confounding variable that you otherwise neglect to measure. 
In consequence, I woudn't say that you have 3 separate result sets and would go for Prof. Harrell's grand model. 
Obviously whether you want to experiment 3 x on 5 mice or 5 x on 3 mice depends on the between cage and between mice variance (plus the fact that 3 x 5 is cheaper than 5 x 3).  
A: The first thing that comes to my mind when I read of the approach that you describe is that there is a miss-match between the idea of replicating an experiment and the use of "success" and "failure" as descriptors of the outcomes. Presumably a success would be a result that is significant in the Neyman-Pearson paradigm (i.e. P < alpha). However, that paradigm allows control of type I errors and specification of power by assuming that the experimenter will act as if the null hypothesis is false when a significant result is observed. In that case there is no point in testing it again--the determination of type I error rates assumes that you discard the hypothesis and move on. Neyman called the process 'inductive behavior'.
A consequence of inductive behavior is that you can't integrate the information across multiple replications of the overall experiment within the Neyman-Pearson paradigm. There would be no sensible interpretation of any sequence of significant and not significant results from repeats of the experiment using the Neyman-Pearson paradigm.
If you use instead Fisher's approach of inductive inference, then you can combine P values from multiple experiments testing the same null hypothesis and obtain a composite value. The P values are indices of the strength of evidence against the null hypothesis (and NOT error rates!) and so, just as it is sensible to combine the evidence from multiple sources before judgement, you should combine the P values. However, while there are several (many?) way to combine the P values, it is perhaps the case that none is perfect.
Arguably the best way to combine evidence from multiple experiments is to use a likelihood-based approach (see Royall 1997 for a very clear exposition) . The likelihood functions for each of the experiments can be combined exactly by multiplication to yield a composite function that can support both interval estimation and statement of the probability of the overall set of results having come from a true null hypothesis.
Royall, R. M. (1997). Statistical evidence: a likelihood paradigm. Chapman & Hall.

It is unfortunate that Fisher was frequently unclear in his expression, and rather inconsistent in his application of the ideas. However, he was undoubtedly in favor of repeating experiments and hated Neyman's idea of inductive behavior: 
“...we may be able to validly apply a test of significance to discredit a hypothesis the expectations from which are widely at variance with the ascertained fact. If we use the term rejection for our attitude to such a hypothesis, it would be clearly understood that no irreversible decision has been taken; that, as rational beings, we are prepared to be convinced by future evidence that appearances were deceptive, and that a remarkable and exceptional coincidence had taken place.” Stat Meth & Sci Inf p.37 (Chapter II section 4).
A: In my case we ran an experiment 5 times in triplicates (to calculate SEM and mean) and selected the most attractive results
