I am about to do a laboratory experiment in the scientific field of soil ecology and hydrology. Beforehand I want to make sure not to make any crucial mistakes, and therefore I would appreciate any hints and comments from your side. The main issue is how to deal with high natural variability (~25%) and a rather small sample size (total max=20).

Short description of the experiment:

We will put soil cores in cylinders and keep them under different moisture scenarios. Three different kind of Carbon forms will be measured for about one month.
We want to know if the variables change within groups and between groups.

The experimental design that was proposed as follows:
There will be 2 different soil types, 2 different treatments + one control, and each treatment should be replicated three times. The total number of cylinders is thus 18 cylinders.

The variability in field measurements can be as high as 25% within one group. Due to pracitcal reasons there cannot be more than 20 cylinders in total.

My questions:

  1. Would it make more sense to have only one treatment and one control, but each one is replicated five times?
  2. Will I be able to draw any reliable conclusions from this kind of experiment under these conditions?
  3. How should I set the parameters to make some power calculations using e.g. G*Power 3? Which test should I choose? What should I set the effect size to and what should the numbers for df be?
  4. How should I analyze the data after the experiment is done? Should I use ANOVA? Can I use a mixed effect model?
  • $\begingroup$ Being not in your area I'm not sure what you mean by "each treatment should be replicated three times" or the precise nature of treatment; are they between or within samples? To me it seems practical to reduce to a treatment and control (after all more samples in a design cell the better), but that is a trade off you should determine based on your needs. Which is more important, experimental power or evaluating both treatments? $\endgroup$ – russellpierce Nov 19 '10 at 16:56

Whether you have a reasonable chance of obtaining (i.e. power to obtain) reliable conclusions depends on how big the effects are you wish to be able to detect. With such small numbers they'll have to be very large. Clearly having fewer treatments and more replications per treatment will give you at least a bit more power, or equivalently the aiblity to detect somewhat smaller effects with the same power.

To put some rough numbers on that, let's ignore the soil types for simplicity (including them will make things more gloomy) and do some standard power calculations two-sample for 2-sample t-tests. If you compare one treatment vs control with 10 in each group (i.e. 20 in total) you'll have 80% power to detect a difference between treatment and control of 1.25 standard deviations (SDs). With two treatments + control, 6 in each group (18 in total), you have 80% power to detect a difference of 1.4 SDs between both treatments combined and control, or 1.6 SDs between either treatment by itself and control (or between the two treatments). It may well be sensible use a log-transform (or perhaps some other transform) your data prior to analysis, in which case the SDs are the SDs of the transformed variables.

In the social sciences, an effect of around 0.8 SDs or over would often be considered "large", and designing a study to detect to have decent power only to detect a bigger effect than this might be politely described as "optimistic". But remember that the SD here is the SD of the residual, unexplained variation. You can reduce this by either (1) making your experimental units more uniform or (2) explaining more of the variation by other means.

  1. The lower the uncontrolled variability the higher the power you'll have to detect effects due to the factors open to experimental manipulation. You say "variability in field measurements can be as high as 25% within one group". But this is a laboratory experiment; is there a reason the variability need be this high in the lab? Can you homogenise your soil before you start the experiment? I guess this may destroy the soil structure though?

  2. Can you take baseline measurements before the treatments are applied? Using these to explain some of the inate variability between units by either analysing change since baseline or (better) adding them to the model as covariates (.e. ANCOVA) may help a lot.

Sorry I haven't mentioned G*Power 3 but i've never heard of it and from a quick look the link you gave it looks considerably more sophisticated, and therefore complicated, than is necessary here.

  • $\begingroup$ Thank you onestop for your detailed answer. I ended up in doing the power analysis with the R function power.t.test. When I do the power test I get somewhat different numbers than you using the following parameters: power.t.test(n=10, power = 0.8, type =c("two.sample")) I get delta=1.32 which is higher than your 1.25 SD's. Am I missing here something? Unfortunately, I can not homogenise the soil before the experiment. I need undisturbed soil columns. I will take baseline measurements before I start the treatment. Any hints on ANCOVA are appreciated. $\endgroup$ – Strohmi Nov 25 '10 at 11:49
  • $\begingroup$ No, you're not missing anything. I was doing the power calc in Stata using its -sampsi- command that uses the normal distribution rather than the t-distribution, so R's power.t.test() function is more accurate, especially when the sample size is very small as here. Power calculations are always approximate, so the difference is rarely important. $\endgroup$ – onestop Nov 25 '10 at 20:16
  • $\begingroup$ I guessed you wouldn't want to homogenise your soil columns. Nevertheless, if would help if you can take more soil columns and use baseline measures to select fairly homogeneous ones to use in the experiment, or find group of 2 or more soil columns that are similar on your three outcome measures and randomly allocate equal numbers within each group to each treatment, i.e. a form of blocking. $\endgroup$ – onestop Nov 25 '10 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.