this is an example experiment design for which I would need some help (the values are the same between assays but should be different).

enter image description here I want to test if the use of drug 1 or drug 2 has a significant effect on the output being measured. The control group was not exposed to any drug. The assay is then repeated 2 more times for statistic significance. Each assay is done at different days with different cells and media.

The problem comes when a statistical test is to be chosen. Here are my thoughts:

  • For each assay individually, ANOVA should not be used because I don't want to compare 3 groups, I want to compare the treated groups with the one control group. So, I would do t-test for Control Vs. D1 and Control vs. D2. Is this correct?

But then, there's the 3 assays considered all together and the representation of the data.

Doing a normalization, with Control being 100% and doing the mean of the 3 assays according to the % is a good way to show the data but statistics wise I don't know how to do this. Pooling all measurements for each group and calculate the mean (doing then the t-tests again) doesn't seem appropriate because even the control group usually shows variations. Normalizing, with Control being 100%, seems even wronger, as the Control group(s) won't have any deviation values.

I thought about introducing the Mean-SD-N for each 3 assays but Graphpad doesn't seem to accept that for t-test and 1wayANOVA.


The stacking approach (pooling 3 experiments) is a bad way to go. You will absolutely have to account for the random, uncontrolled variation between experiments. Even if the protocol were followed to a T, who knows what differences in atmospheric pressure might have affected your mass spec's calibration. Control of variation can be achieved in a number of ways.

The heirarchical approach would involve calculating test statistics for each experiment and combining them using weighting to obtain a pooled statistic. This is similar to a meta-analysis and makes few assumptions about the variance between samples.

The mixed effect approach involves using either a MANOVA or repeated measures ANOVA by using fixed or random effects indicating the experimental iteration. I would strongly favor the fixed effect approach because, while it takes a toll on the effective degrees of freedom for the experiment, it handles the between-experiment variation very efficiently. (Whereas random effect inference would depend upon an n=3 estimate of a normal random effect variance which is unstable indeed).

Secondly, I take issue with your idea to combine treatment groups. Without knowing dosing, or methods, one cannot assume the 1:1:1 assignment to trt1 / trt2 / ctl is in any way representative of a meaningful group when you combine trt1:trt2. This is not a balanced design issue, but the fact that the rather strong assumptions about trt1 and trt2 being a single treatment make inference extremely hard to interpret. If, on the other hand, trt1 and trt2 corresponded to different dosages of the same drug (or could be combined in a logical way), it could be valid to assign a single treatment variable, continuous, with 3 levels for 0 (control) trt1 dosage and trt2 dosage.

If your goal, however, is to test the hypothesis "H_0: treatment does not yield response" vs "H_1: at least 1 treatment yields a response different than control". You can use a likelihood ratio test to fit the "full" model (with treatment effects for trt1 and trt2) versus the "reduced" model (with no treatment variable) and conduct the 2 degree of freedom likelihood ratio test. I'm not too fond of the interpretation of those results, but it does afford you a lot of power to detect differences simultaneously, relative to conducting two tests: trt1 vs ctl and trt2 vs ctl and handling their multiplicity using Bonferroni correction.

PS before I get rained upon with "gentle" correlation/causation reminders, I'll note that I used the causal statement of hypothesis because the question deeply suggested this was some kind of in vitro (efficacy) test.

  • $\begingroup$ First of all, thank you very much for a detailed answer. Secondly, maybe I should immediately provide some more information. - It is an in vitro assay, the effect of a drug in cell counts. - Maybe I didn't explain myself correctly but my intention is only to see if any of the drugs as an effect. Either one or both but they are completely independent. They are not different doses of the same and I'm not interested in combining the treatment groups. $\endgroup$ – user34963 Nov 18 '13 at 17:46
  • $\begingroup$ What I meant was to combine the 6+6+6 measurements of each assay, in order to perform a final test which takes in consideration all the measurements of the 3 assays together. The final questions being always the same: Is the Control group different from D1 group? and Is the Control group different from D2 group? $\endgroup$ – user34963 Nov 18 '13 at 17:47
  • $\begingroup$ Well I've set out the methods for combining multiple experiments' data, and I've also mentioned the likelihood ratio test as a method for joint inference on trt2 / trt1 / ctl. You can alternately structurally test D1 vs ctl and D2 vs ctl, but you need to handle multiple testing. $\endgroup$ – AdamO Nov 18 '13 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.