I have data from an experiment comparing plant weights for 4 independent treatment groups. The data seem to be normally distributed (I have been warned about using statistical tests for normality). However, my sample size is really small ($n=16$ in total; $n=4$ per group). Would a Kruskal-Wallis test be more robust to small sample sizes, or would the ANOVA be equally as robust?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ sorry for not specifying - each group has 4 plants (very small i know but larger sample sizes were not possible) $\endgroup$– user411569Commented Apr 25 at 19:42
-
$\begingroup$ 1. What thing are you concerned about robustness of (significance level? power?) and in relation to what? potential misidentification of the conditional distribution of the response? or something else? $\,$ 2. How are you concluding that you have normality with n=4? Are you using some external information about the distribution within one or more groups? Or are you examining residuals from some fit? (Even so, 16 residuals - with fewer degrees of freedom - may not tell you a lot). $\endgroup$– Glen_bCommented Apr 25 at 22:30
-
$\begingroup$ Take a look at the marginal distribution of internally standardized values with n=4 (say) from various distributions for example. $\endgroup$– Glen_bCommented Apr 25 at 22:36
-
$\begingroup$ For future research; Once the size of an effect is assumed, you can calculate the necessary sample size for subsequent research and roughly determine how your experiment will be set up. This could help everyone understand why so many samples are needed and prevent conversations about how to obtain more data. $\endgroup$– MarcoCommented Apr 26 at 6:58
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
"Robust to small sample sizes" doesn't really mean anything. The sample size isn't an assumption of any test. With very few data, you are likely to have very low power. Below a certain $n$, the KW test won't be able to return a significant result, even if the groups do not overlap at all:
kruskal.test(list(1:2,3:4,5:6,7:8))
# Kruskal-Wallis rank sum test
#
# data: list(1:2, 3:4, 5:6, 7:8)
# Kruskal-Wallis chi-squared = 6.6667, df = 3, p-value = 0.08332
summary(aov(c(1:2,3:4,5:6,7:8)~letters[c(1,1,2,2,3,3,4,4)]))
# Df Sum Sq Mean Sq F value Pr(>F)
# letters[c(1, 1, 2, 2, 3, 3, 4, 4)] 3 40 13.33 26.67 0.00418 **
# Residuals 4 2 0.50
With $n=4$ per group, a KW test can return a significant result, but power is likely to be low. An ANOVA may have more power, but the validity of the result will be heavily dependent on the normality of the population. Unfortunately, you are on the horns of a dilemma. If you wanted to try an ANOVA, I would not test the data for normality, but you could look at a qq-plot of the residuals. More generally for situations like yours, it is considered best to just go with the KW and stick with it, whether it is significant or not.
answered Apr 25 at 20:13
-
2$\begingroup$ "the validity of the result will be heavily dependent on the normality of the population" - if this were so, it could never be used in a valid manner as nothing ever is really normally distributed. $\endgroup$ Commented Apr 25 at 22:51
-
1$\begingroup$ That is literally true, @ChristianHennig, but data can well be 'normal enough' that such tests are OK. It seems dicey to me here, but it might be OK. $\endgroup$ Commented Apr 26 at 0:39
-
$\begingroup$ The way I'd contextualize the choice between KW and ANOVA in this case is that if one has reason to believe within-group observations are well-modeled by a normal distribution (e.g., available historical data) then ANOVA could be justified even for a small sample size, but any resulting inferences would necessarily only be interpretable under this assumption. $\endgroup$– heropupCommented Apr 26 at 6:04
-
$\begingroup$ @gung-ReinstateMonica As I argue in my answer, my point is not that such a test cannot be used because data are not normal (although whether data are "approximately normal" and whether ANOVA will be OK are not exactly the same thing). My point is rather that as statisticians we shouldn't tell others that data need to be normal if in fact this isn't true. $\endgroup$ Commented Apr 26 at 8:53
$\begingroup$
$\endgroup$
A general remark regarding model assumptions: Statistical model assumptions are idealisations and are never true (George Box wrote: "All models are wrong but some are useful"). Your data are not normally distributed as no data ever are.
Generally, as written already by gung - Reinstate Monica, Kruskal-Wallis with very small sample sizes isn't very good, because it has a very low power. The ANOVA will be OK if the group means and the pooled variance within treatments on which the ANOVA is based represent well the structure of the data. In order to assess this I'd plot all the data in a dot plot by group like this (if you don't have an overplotting issue, you don't need to shift some points in the same group to the left and some to the right). Reasons for concern would be extreme outliers, very strong differences between within-group variances, maybe issues with skewness (although with 4 observations per group it is hard to imagine really bad situations regarding skewness).
The normality assumption means that the null hypothesis assumes normality, and if you don't reject it, then the observed test statistic is compatible with equal normal distributions, which has the usual interpretation of insignificance regardless of whether data are truly normal or not.
If you reject, the data are incompatible with equal normal distributions, but one may question whether the data are compatible with any set of distributions for the four groups that could be interpreted as all groups tending to produce the same sizes of values on average. Depending on how your data look like, it may be pretty clear that they aren't. Generally in a situation like this it is hard to set up equal distributions that tend to reject equality much more easily than the normal, meaning that chances are a rejection can be interpreted as you think it should. I'd say that if you get a result that (a) isn't borderline (such as $p=0.042$) and (b) clearly corresponds to the impression that you get when looking at the data, feel free to interpret it accordingly.
