# Significance test with non-normal, bounded data?

I am attempting to do a one-sample significance test to determine whether a set of data differs from a given value (0 in this case). The issues I have with these data:

• Non-normally distributed data, which is also bounded from -1 to 1
• Small sample sizes (from about 5-20)

Here's an example of what my data looks like: (-0.2, -0.05, 0.1, 0.15, 0.2) I want to test whether this group of 5 points differs significantly from 0. My default for running this test would be a one-sample Student's t-test, with mu=0. When I do that for this set, I get p=0.61, meaning this set is not different from mu (0).

However, I'm concerned that because of the small sample size, possibility for non-normal distributions, and the fact that my data is bounded from -1 to 1, this t-test may not be the most appropriate test.

Do you think it's appropriate, or have a suggestion of a better test? Maybe a Wilcoxon signed rank test?

* More background on the data if necessary: Basically I'm doing a meta-analysis. I'm looking at results from multiple models, and calculating the mean result from each model. In this case 1 indicates only increasing results and -1 indicates only decreasing results, so that's why data is bound from -1 to 1. *

• A (nonparametric) one-sample 'Wilcoxon signed-rank test would be OK for the non-normal data you describe, but probably won't often give significant results for samples as small as $n = 5$ (as in your example). – BruceET Jun 17 '19 at 22:47
• Ok, thanks for the input. Mostly I have n~10, so it shouldn't be a big problem. I ran a few examples and the Wilcoxon seemed a little more conservative than the t-test, but not very different. – Scott Z Jun 18 '19 at 0:03
• I was just going to append a note to my answer. I ran several simulations with t.tests and found no serious difficulties. With $n = 10$ it is stretching the legendary robustness of the t test a bit far to use it for your data. If you are writing a final project report, thesis, or paper for publication, you might do preliminary analyses with t tests and then check results of significant cases using Wilcoxon. Then to keep reviewers happy, briefly mention Wilcoxon results in footnotes. – BruceET Jun 18 '19 at 0:10
• Your data are discrete; indeed if you assume independence and a common probability of a 1 within each average, it'd be a scaled binomial. If the means are not getting close to either boundary the t-test probably will be okay, as long as it would be otherwise suitable (which is not completely clear to me). I am not clear on why you're averaging the 1's and -1's instead of analyzing them as is. – Glen_b Jun 18 '19 at 6:57

Wilcoxon signed-rank test for your data. For the data in your example, a one-sample Wilcoxon test (of the null hypothesis that the population median is $$\eta = 0)$$ gives the following non-significant result (in R):

x = c(-0.2, -0.05, 0.1, 0.15, 0.2)
wilcox.test(x)

Wilcoxon signed rank test with continuity correction

data:  x
V = 9.5, p-value = 0.6845
alternative hypothesis: true location is not equal to 0

Warning message:
In wilcox.test.default(x) : cannot compute exact p-value with ties


According to this procedure, you have a tie because in the initial ranking step signs are ignored, so that $$pm 0.2$$ counts as a tie. But even with one tie, the P-value is not likely to be catastrophically wrong.

Difficulties with very small samples. You could get a significant result for a one-sided test (of $$H_0: \eta =0$$ vs. $$H_a: \eta > 0$$ with $$n= 5$$ if all five observations are either above or below $$0.$$ for example:

x = c(0.1, 0.3, 0.4, 0.5, 0.8)
wilcox.test(x, alt="gr")$p.val [1] 0.03125  Power of Wilcoxon SR test with 15 observations. Here is a simulation with a million samples of size $$n = 15$$ from $$\mathsf{Unif}(-.4, 1)$$ rounded to one place, so that the population median is $$\eta= 0.7.$$ If we have such a sample, what is the probability that $$H_0: \eta = 0$$ will be rejected in favor of $$H_a: \eta > 0\,?$$ This probability is called the 'power' of the test. The simulation suggests that the power is about 90%. (There are warning messages; with 15 observations rounded to one place, there are almost sure to be ties, but we ignore error messages. A run with no rounding, hence no ties, gave almost the same result.) set.seed(617) pv = replicate( 10^5, wilcox.test(round(runif(15, -.4, 1),1), alt="g")$p.val )
There were 50 or more warnings (use warnings() to see the first 50)
mean(pv <= .05)
[1] 0.81138


The first of the one million runs gave the following result, of which we used only the P-value in the simulation:

set.seed(617)
x = round(runif(15, -.4, 1),1)
sort(x)
[1] -0.4 -0.3 -0.1 -0.1  0.1  0.2  0.3  0.5  0.7
[10]  0.7  0.8  0.8  0.8  1.0  1.0

wilcox.test(x, alt="g")

Wilcoxon signed rank test with continuity correction
data:  x
V = 103.5, p-value = 0.007189
alternative hypothesis: true location is greater than 0

Warning message:
In wilcox.test.default(x, alt = "g") :
cannot compute exact p-value with ties


Summary. The one-sample Wilcoxon test seems best for data, with a couple of caveats: (a) If you can use samples larger than about $$n = 10$$ you will have a better chance of discovering when your observations are not centered at $$0.$$ (b) You might get fewer ties (thus potentially somewhat more reliable results) if you can use data with two (or more) decimal places rather than one.

• Thank you for your detailed answer. Luckily, my real data has many decimal places, so that should reduce the ties. Would those make a big impact? The Wilcoxon test certainly seems more conservative, but that may be a good thing. I just looked at an instance with only 3 observations - even if these are all negative and pretty far from 0 (-0.54, -0.23, -0.51), I'm getting p=0.25 with Wilcox. I get p=0.05 with a t-test in that case. However, with only 3 observations it seems fishy to ascribe significance anyway. – Scott Z Jun 18 '19 at 17:40
• The distribution theory for Wilcoxon's rank-based tests becomes more complicated when ties are present. Most modern computer implementations, such as the one in R do a good job of giving accurate approximate P-values. So I don't expect huge differences by avoiding ties. For $n < 5,$ Wilcoxon SR test is essentially useless. (Even with $n=5,$ P-value of two-sided test can never be smaller than $1/16 > .05.)$ Reducing data to ranks loses information, so if data are normal Wilcoxon SR test cannot generally have a large a power as a t test, which is exact for normal data. – BruceET Jun 18 '19 at 17:58