I would like to ask for help in choosing the correct statistical test for analyzing experimental data on small samples (4 repeats)

THE EXPERIMENT: the experiment simulates development of a deposit on a sample. It allows for evaluation of different deposit sources and factors affecting its growth. Each combination of source & factor is a treatment. The main response is the deposit extent recorded 4 times over the experiment duration.


  • 18 treatments
  • 4 repeats for each treatment
  • All the setups are tested at the same time (18 treatments x 4 = 72 setups), under the same, controlled conditions
  • the test goes for 4 weeks and responses are collected 4 times for every single setup

I choose 4 repeats, because the reference setups show high repeatability, and because of practical limitations. However, for a few of the treated setups I have identified single outliers, based on visual data analysis. So for some setups I have only 3 repeats available.

THE DATA ANALYSIS: The main aim is to compare the final deposit extent and its growth trend (e.g. inhibited after 1 week vs still developing). I was considering the following approaches:

  • DOE in SAS JMP (regression on dummy categorical variables), but it always asks for randomization, while all my setups are run at the same time
  • ANOVA, but I was advised to use it only for much larger sample sizes (>30)
    • Recent idea: compare data visually based on graph with 95% confidence intervals using a Student's t distribution (CONFIDENCE.T function in Excel)
  • 1
    $\begingroup$ (1) So for some treatments you've discarded one quarter of your observations just because they were higher or lower than you thought they should be compared to the other observations for that treatment? (2) "Randomization" refers to the allocation of treatments, to whatever "set-ups" are, & doesn't necessarily imply a sequence in time. (3) It's unclear what the difference is between the first two approaches you've considered - ANOVA's equivalent to a regression on categorical predictors represented by dummy variables. (4) Some explanation of the nature of the experiment & the ... $\endgroup$ – Scortchi Nov 17 '15 at 9:37
  • $\begingroup$ ... measurements, & a more precise statement of the goals of your analysis might help potential answerers. $\endgroup$ – Scortchi Nov 17 '15 at 9:58
  • $\begingroup$ h) (1) I couldn't apply any mathematical formula for outliers treatment due to such a small sample size. Nevertheless I have discarded single results only in case where 3 setups show very high repeatability and a single one was is way off, what I attribute to e.g. a setup flaw. (2) I didn't identify any step in the allocation of treatment, where order could be of importance, and often treatments were simultaneous $\endgroup$ – jacek Nov 17 '15 at 10:13
  • $\begingroup$ (3) I almost agree with you, however after hours spent on reading on "regression vs anova" I find that in many instances people address these methods to different problems, and state than one fits better than another in certain situations, (some refer to different ways of null hypothesis definition). I must admit that I was first tempted by DOE, as I find that most case studies which we analyzed during a DOE course were based on single measurements or just 2 repeats per treatment (only later I started to learn on sample size calculation). $\endgroup$ – jacek Nov 17 '15 at 10:13
  • $\begingroup$ (1) It'd be advisable to, at the least, (a) compare results with & without those "outliers" included. (b) examine the set-ups for evidence of a flaw rather than assuming it. (2) Again, "randomization" isn't just, or even mainly, concerned with order in time, but with any potential differences between set-ups that might confound estimation of treatment effects. (3) Try both & reconcile the results. And "DOE" stands for "Design of Experiments", experiments typically analysed by regression/ANOVA. $\endgroup$ – Scortchi Nov 17 '15 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.