How to setup a laboratory experiment in Ecological Research under high natural variability

Question

I am about to do a laboratory experiment in the scientific field of soil ecology and hydrology. Beforehand I want to make sure not to make any crucial mistakes, and therefore I would appreciate any hints and comments from your side. The main issue is how to deal with high natural variability (~25%) and a rather small sample size (total max=20).

Short Description of the Experiment:

We will put soil cores in cylinders and keep them under different moisture scenarios. Three different kind of Carbon forms will be measured for about one month.
We want to know if the variables change within groups and between groups.

The experimental design that was proposed is as follows:
There will be 2 different soil types, 2 different treatments + one control, and each treatment should be replicated three times. The total number of cylinders is thus 18 cylinders.

The variability in field measurements can be as high as 25% within one group. Due to pracitcal reasons there can not be more than 20 cylincers in total.

My Questions:

1. Would it make more sense to have only one treatment and one control, but each one is replicated five times?
2. Will I be able to draw any reliable conclusions from this kind of experiment under these conditions?
3. How should I set the parameters to make some power calculations using e.g. G*Power 3? Which test should I choose? What should I set the effect size to and what should the numbers for df be?
4. How should I analyze the data after the experiment is done? Should I use ANOVA? Can I use a mixed effect model?

Answer 1

Whether you have a reasonable chance of obtaining (i.e. power to obtain) reliable conclusions depends on how big the effects are you wish to be able to detect. With such small numbers they'll have to be very large. Clearly having fewer treatments and more replications per treatment will give you at least a bit more power, or equivalently the aiblity to detect somewhat smaller effects with the same power.

To put some rough numbers on that, let's ignore the soil types for simplicity (including them will make things more gloomy) and do some standard power calculations two-sample for 2-sample t-tests. If you compare one treatment vs control with 10 in each group (i.e. 20 in total) you'll have 80% power to detect a difference between treatment and control of 1.25 standard deviations (SDs). With two treatments + control, 6 in each group (18 in total), you have 80% power to detect a difference of 1.4 SDs between both treatments combined and control, or 1.6 SDs between either treatment by itself and control (or between the two treatments). It may well be sensible use a log-transform (or perhaps some other transform) your data prior to analysis, in which case the SDs are the SDs of the transformed variables.

In the social sciences, an effect of around 0.8 SDs or over would often be considered "large", and designing a study to detect to have decent power only to detect a bigger effect than this might be politely described as "optimistic". But remember that the SD here is the SD of the residual, unexplained variation. You can reduce this by either (1) making your experimental units more uniform or (2) explaining more of the variation by other means.

1. The lower the uncontrolled variability the higher the power you'll have to detect effects due to the factors open to experimental manipulation. You say "variability in field measurements can be as high as 25% within one group". But this is a laboratory experiment; is there a reason the variability need be this high in the lab? Can you homogenise your soil before you start the experiment? I guess this may destroy the soil structure though?

2. Can you take baseline measurements before the treatments are applied? Using these to explain some of the inate variability between units by either analysing change since baseline or (better) adding them to the model as covariates (.e. ANCOVA) may help a lot.

Sorry I haven't mentioned G*Power 3 but i've never heard of it and from a quick look the link you gave it looks considerably more sophisticated, and therefore complicated, than is necessary here.