# Which test for small sample sizes?

I have got a total sample size of 12 soil samples which were split into 4 groups of $n=3$. A number of different variables were measured like content of organic carbon. These data are logistically difficult and expensive to collect, so collecting more data is not possible. Now I want to compare the means of the 4 groups ($n=3$) by testing. But there is the problem that the normal distribution (Shapiro-Wilk) and homoscedasticity (Brown-Forsythe/Levene) is not given in each sample. So which test should I prefer? What do you think? Is it even useful to use test statistics in this case? Are there any alternatives beneath descriptive statistics?

OK, here are some details. The soil samples are from a long-term experiment site with four different treatments (plough, no tillage and so on). From each of these four treatments I have 3 soil samples. The different variables are measured in metric data. For example - content of organic carbon [g/kg]: (Group 1: 12.1 // 13.2 // 13.5), (Group 2: 13.1 // 13.9 // 13.5), (Group 3: 8.9 // 10.2 // 11.9). Now I wanted to test if there are any significant differences among the medians of these groups. In addition, concentrations of different substances in soils are generally not normal distributed.

• what kinds of variables did you collect data on? What types of variables (ie. continuous or binary/categorical)? Commented Sep 3, 2013 at 22:29
• I made a suggestion before relating to permutation-based methods; those suffer from exactly the same problem that I point out in respect of rank based methods below. Commented Sep 3, 2013 at 22:55

Please refer to questions on testing for assumptions of your model. They are usually a pointless endeavour and particularly so in this case.

Simple parametric tests, such as t-test and ANOVA, were designed for small samples sizes. So, there's not necessarily any good reason to be considering other tests. But one would really need much more information to offer better advice.

EDIT: You have 9 data points in total. Just talk about the raw data and ignore statistical summaries. The issue with the fundamental assumption of representativeness, for any discussion of the data, is going to be your biggest problem. Non-parametric tests require bigger Ns than you have and parametric ones require more information. From the values you have I'd just say that groups 1 and 2 are easily samples from the same soil but for group 3 it might be a little less and argue for collecting more data.

• +1 The OP may have some idea of suitable assumptions from external-to-the-data considerations - e.g. other, similar, data from other studies might give an idea of whether the distributions would tend to be skew, for example, with nonconstant variance -- or theoretical considerations may give some feel for things like that. If the OP feels that the usual t-test assumptions aren't reasonably close to tenable, there's always parametric methods that make different assumptions. ...(ctd) Commented Sep 3, 2013 at 23:02
• (ctd)...Conveniently, for example, inverse Gaussian, gamma and in some packages, Tweedie distribution families can be accommodated in the GLM framework, giving some useful scope for dealing with comparisons of means in the presence of moderate to quite heavy skewness and mean-related heteroskedasticity. Commented Sep 3, 2013 at 23:02
• Yeah, maybe but the OP clearly states they can't get the tests on the samples. You may be right but that brings us back to...we need more info.
– John
Commented Sep 4, 2013 at 0:42
• Yes, there's no point even trying to assess the suitability of the assumptions on the OP's own data. My first sentence referred to 'external to the data' considerations such as other studies for precisely that reason. Commented Sep 4, 2013 at 0:49
• Perhaps we were talking cross purposes. If you're suggesting the OP could gather this useful information I agree. I was merely saying that, from reading what he/she said, it looks like they haven't done that yet.
– John
Commented Sep 4, 2013 at 1:30

Basically, any sort of hypothesis test based on very small samples requires strong assumptions. In the frequentist framework these are basically made in regards to the distribution of your data (properties like normality, homoscedasticity, etc.) which you can't check in small samples and also because the robustness properties that many tests thankfully possess are not yet in force.

Note that with check I mean more things like visually inspecting the distribution instead of formal hypothesis testing for assumption which is not generally useful as John states. In the bayesian framework some of these assumptions can also be in the form of prior information.

As a very general advise it is often necessary and even useful to make strong assumption like these above but they should a) be stated clearly, b) discussed whether they are reliable (e.g. give references) and c) ideally some indication of how much your results depend on these assumptions.

More specific for your question I am surprised that the shapiro-test is actually able to find deviations given your small sample size of 3 samples per group. Are you sure you used it correctly? It is within-group normality that is required not overall normalness which is a common mistake for beginner's to make. For example, from your given organic carbon content data I would say that it's impossible to say whether this data is from a normal distribution or not. One starting point before doing an analysis is always to try to get an impression of what distribution you can expect from your data based on previous samples by other peoples.