6
$\begingroup$

Due to limitations in experimental setup, I only have small data sets with n=3. Despite the low df the difference between treated and control is large enough to generate a significant p-value.

The problem is that with small sample sizes doing a t-test becomes more sensitive to the assumption that the data are drawn from a population of a normal distribution. In my case however, multiple independent experiments consistently yield a similar result.

I cannot group the data of the experiments because of small differences in the data. For example the increase between treated and control in one experiment is slightly bigger than in another experiment, which is likely caused by small variances in experimental conditions (it would get technical to explain this further). Despite this the same increase is consistently observed and each time the 3 data points have a small standard deviation for both groups.

So my question is whether it is defensible to make the normality assumption based on the data of multiple independent experiments with a small sample size? If not am I right that it would not be appropriate to use any statistics in this case?

Improve this question
$\endgroup$
6
$\begingroup$

This may help:

DR Cox, PJ Solomon. 1986. Analysis of variability with large numbers of small samples. Biometrika 73: 543-554.

Abstract: Procedures are discussed for the detailed analysis of distributional form, based on many samples of size r, where especially r= 2, 3, 4. The possibility of discriminating between different kinds of departure from the standard normal assumptions is discussed. Both graphical and more formal procedures are developed and illustrated by some data on pulse rates.

Improve this answer
$\endgroup$
  • $\begingroup$ Unfortunately my university library won't let me open the article. Looks interesting though, any other way I could get access to it? $\endgroup$ – user26437 Jun 4 '13 at 1:10
  • 1
    $\begingroup$ Email the authors for a reprint or buy with USD25 from biomet.oxfordjournals.org/content/73/3/543.full.pdf+html $\endgroup$ – Nick Cox Jun 4 '13 at 7:37
3
$\begingroup$

Due to limitations in experimental setup, I only have small data sets with n=3. Despite the low df the difference between treated and control is large enough to generate a significant p-value.

The problem is that with small sample sizes doing a t-test becomes more sensitive to the assumption that the data are drawn from a population of a normal distribution. In my case however, multiple independent experiments consistently yield a similar result.

I cannot group the data of the experiments because of small differences in the data.

If you treat the experiments as blocks, that can be used to account for this. (Alternatively, you may want to use random effects term on intercepts, especially if this is not under control.)

So my question is whether it is defensible to make the normality assumption based on the data of multiple independent experiments with a small sample size?

You can attempt to assess it if you assume a common error distribution and combine residuals across all experiments in order to do say a normal Q-Q plot (normal scores plot).

If not am I right that it would not be appropriate to use any statistics in this case?

You can still test your hypothesis without normality, but beware, you won't get very significant results with nonparametric tests and such tiny sample sizes.

However, the combining experiments strategy (of using blocks) can work there too.

Improve this answer
$\endgroup$
  • $\begingroup$ Wilcoxon wouldn't be appropriate because the data within one experiment aren't paired. But if I understand you correctly you mean grouping the results of one experiment into a block and then doing a Wilcoxon test to compare paired blocks? $\endgroup$ – user26437 Jun 4 '13 at 1:19
  • $\begingroup$ Sorry if I'm slow to follow - I'm quite new to statistics. A Mann Whitney still gives me p values of >0.005 so I'm not very worried about significant results. The data are far apart and it's a consistent so I'm confident it's a significant effect, I just want to make sure I choose the right test. $\endgroup$ – user26437 Jun 4 '13 at 1:26
  • $\begingroup$ The blocking would make it Friedman (though that's perhaps not the only possibility) not Wilcoxon. The data within experiments are dependent because of the differences between experiments. $\endgroup$ – Glen_b Jun 4 '13 at 3:14
2
$\begingroup$

There are certainly alternative statistics. You could do permutation tests, for example. You could also do nonparametric tests, such as Wilcoxon.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.