testing that mean(x) = k with a small sample and unknown distribution of x

Suppose a number of random variables (call them "errors") observed for different "scores". For every score 11 errors have been observed. See the plot below to get the picture:

I would like to know in what scores the error is "big". For this, I think a good approach would be to test the hypothesis

$\text{H}_0: \overline{\text{error}} = 0$

$\text{H}_1: \overline{\text{error}} \neq 0$

where $\overline{\text{error}}$ is the mean error, for each score. However, there are only 11 errors for each score and their distribution is unknown. What test (if any) would be adequate in such circumstances?

-
Just a word of caution: some of the scores have much lower variability associated with them, so any test of mean would have a much higher power. You might wind up rejecting the null hypothesis for the low-variability groups, because the mean is slightly different from 0, while there is almost no chance of rejection with the high variability groups unless the bias is glaring. –  Aniko Apr 19 '12 at 14:55
Right. This is one of the reasons why confidence intervals are better than hypothesis tests in general. –  dominic999 Apr 19 '12 at 21:39

I would NOT use the t-test in this case. The confidence intervals it gives depend on the assumption that the sample mean is approximately normal, which means either the errors are normally distributed or your sample size is large enough that the Central Limit Theorem kicks in. With a sample size of 11 I would try to assume neither without further knowledge. You could easily fail to distinguish a log-normal distribution from a normal one, just to give an example.

If the error variation would be the same across scores you might have tried some sort of regression model and then would have been able to check the residuals for normality with some accuracy. But the error variation is clearly not the same across samples.

The easiest thing would be to abandon the mean and use the median, which I think is better suited for your case anyway. In this case you can use a one-sample Wilcoxon Test (wilcox.test in R) in good conscience.

-

If I understand your situation correctly, you would simply use a t-test. Or better, construct confidence intervals around your means. This is easy:

lower bound = mean(error) - t.crit*[(st.dev(error)/sqrt(11)]
upper bound = mean(error) + t.crit*[(st.dev(error)/sqrt(11)]


You need to find the proper value for t.crit; in this situation (as given by R) it is:

> qt(0.975,11)
[1] 2.200985

-
thanks for the answer, though I think I'll use the Wilcoxon test because it makes less assumptions –  Ernest A Apr 19 '12 at 16:49
No prob! Note that although it's true the t-test assumes normality, it is also robust to minor deviations, and seeing your boxplots, there's no clear evidence of non-normality. So I'd feel totally comfortable using the t procedure. –  dominic999 Apr 19 '12 at 21:49