Which test for small sample sizes?

Question

I have got a total sample size of 12 soil samples which were split into 4 groups of $n=3$. A number of different variables were measured like content of organic carbon. These data are logistically difficult and expensive to collect, so collecting more data is not possible. Now I want to compare the means of the 4 groups ($n=3$) by testing. But there is the problem that the normal distribution (Shapiro-Wilk) and homoscedasticity (Brown-Forsythe/Levene) is not given in each sample. So which test should I prefer? What do you think? Is it even useful to use test statistics in this case? Are there any alternatives beneath descriptive statistics?

---

OK, here are some details. The soil samples are from a long-term experiment site with four different treatments (plough, no tillage and so on). From each of these four treatments I have 3 soil samples. The different variables are measured in metric data. For example - content of organic carbon [g/kg]: (Group 1: 12.1 // 13.2 // 13.5), (Group 2: 13.1 // 13.9 // 13.5), (Group 3: 8.9 // 10.2 // 11.9). Now I wanted to test if there are any significant differences among the medians of these groups. In addition, concentrations of different substances in soils are generally not normal distributed.

Answer 1

Please refer to questions on testing for assumptions of your model. They are usually a pointless endeavour and particularly so in this case.

Simple parametric tests, such as t-test and ANOVA, were designed for small samples sizes. So, there's not necessarily any good reason to be considering other tests. But one would really need much more information to offer better advice.

EDIT: You have 9 data points in total. Just talk about the raw data and ignore statistical summaries. The issue with the fundamental assumption of representativeness, for any discussion of the data, is going to be your biggest problem. Non-parametric tests require bigger Ns than you have and parametric ones require more information. From the values you have I'd just say that groups 1 and 2 are easily samples from the same soil but for group 3 it might be a little less and argue for collecting more data.

Answer 2

Basically, any sort of hypothesis test based on very small samples requires strong assumptions. In the frequentist framework these are basically made in regards to the distribution of your data (properties like normality, homoscedasticity, etc.) which you can't check in small samples and also because the robustness properties that many tests thankfully possess are not yet in force.

Note that with check I mean more things like visually inspecting the distribution instead of formal hypothesis testing for assumption which is not generally useful as John states. In the bayesian framework some of these assumptions can also be in the form of prior information.

As a very general advise it is often necessary and even useful to make strong assumption like these above but they should a) be stated clearly, b) discussed whether they are reliable (e.g. give references) and c) ideally some indication of how much your results depend on these assumptions.

More specific for your question I am surprised that the shapiro-test is actually able to find deviations given your small sample size of 3 samples per group. Are you sure you used it correctly? It is within-group normality that is required not overall normalness which is a common mistake for beginner's to make. For example, from your given organic carbon content data I would say that it's impossible to say whether this data is from a normal distribution or not. One starting point before doing an analysis is always to try to get an impression of what distribution you can expect from your data based on previous samples by other peoples.