testing that mean(x) = k with a small sample and unknown distribution of x

Question

Suppose a number of random variables (call them "errors") observed for different "scores". For every score 11 errors have been observed. See the plot below to get the picture:

I would like to know in what scores the error is "big". For this, I think a good approach would be to test the hypothesis

$\text{H}_0: \overline{\text{error}} = 0$

$\text{H}_1: \overline{\text{error}} \neq 0$

where $\overline{\text{error}}$ is the mean error, for each score. However, there are only 11 errors for each score and their distribution is unknown. What test (if any) would be adequate in such circumstances?

Answer 1

I would NOT use the t-test in this case. The confidence intervals it gives depend on the assumption that the sample mean is approximately normal, which means either the errors are normally distributed or your sample size is large enough that the Central Limit Theorem kicks in. With a sample size of 11 I would try to assume neither without further knowledge. You could easily fail to distinguish a log-normal distribution from a normal one, just to give an example.

If the error variation would be the same across scores you might have tried some sort of regression model and then would have been able to check the residuals for normality with some accuracy. But the error variation is clearly not the same across samples.

The easiest thing would be to abandon the mean and use the median, which I think is better suited for your case anyway. In this case you can use a one-sample Wilcoxon Test (wilcox.test in R) in good conscience.

Answer 2

If I understand your situation correctly, you would simply use a t-test. Or better, construct confidence intervals around your means. This is easy:

lower bound = mean(error) - t.crit*[(st.dev(error)/sqrt(11)]
upper bound = mean(error) + t.crit*[(st.dev(error)/sqrt(11)]

You need to find the proper value for t.crit; in this situation (as given by R) it is:

> qt(0.975,11)
[1] 2.200985