0
$\begingroup$

I am attempting to do a one-sample significance test to determine whether a set of data differs from a given value (0 in this case). The issues I have with these data:

  • Non-normally distributed data, which is also bounded from -1 to 1
  • Small sample sizes (from about 5-20)

Here's an example of what my data looks like: (-0.2, -0.05, 0.1, 0.15, 0.2) I want to test whether this group of 5 points differs significantly from 0. My default for running this test would be a one-sample Student's t-test, with mu=0. When I do that for this set, I get p=0.61, meaning this set is not different from mu (0).

However, I'm concerned that because of the small sample size, possibility for non-normal distributions, and the fact that my data is bounded from -1 to 1, this t-test may not be the most appropriate test.

Do you think it's appropriate, or have a suggestion of a better test? Maybe a Wilcoxon signed rank test?

* More background on the data if necessary: Basically I'm doing a meta-analysis. I'm looking at results from multiple models, and calculating the mean result from each model. In this case 1 indicates only increasing results and -1 indicates only decreasing results, so that's why data is bound from -1 to 1. *

Improve this question
$\endgroup$
4
  • $\begingroup$ A (nonparametric) one-sample 'Wilcoxon signed-rank test would be OK for the non-normal data you describe, but probably won't often give significant results for samples as small as $n = 5$ (as in your example). $\endgroup$ – BruceET Jun 17 '19 at 22:47
  • $\begingroup$ Ok, thanks for the input. Mostly I have n~10, so it shouldn't be a big problem. I ran a few examples and the Wilcoxon seemed a little more conservative than the t-test, but not very different. $\endgroup$ – Scott Z Jun 18 '19 at 0:03
  • $\begingroup$ I was just going to append a note to my answer. I ran several simulations with t.tests and found no serious difficulties. With $n = 10$ it is stretching the legendary robustness of the t test a bit far to use it for your data. If you are writing a final project report, thesis, or paper for publication, you might do preliminary analyses with t tests and then check results of significant cases using Wilcoxon. Then to keep reviewers happy, briefly mention Wilcoxon results in footnotes. $\endgroup$ – BruceET Jun 18 '19 at 0:10
  • $\begingroup$ Your data are discrete; indeed if you assume independence and a common probability of a 1 within each average, it'd be a scaled binomial. If the means are not getting close to either boundary the t-test probably will be okay, as long as it would be otherwise suitable (which is not completely clear to me). I am not clear on why you're averaging the 1's and -1's instead of analyzing them as is. $\endgroup$ – Glen_b Jun 18 '19 at 6:57
0
$\begingroup$

Wilcoxon signed-rank test for your data. For the data in your example, a one-sample Wilcoxon test (of the null hypothesis that the population median is $\eta = 0)$ gives the following non-significant result (in R):

x = c(-0.2, -0.05, 0.1, 0.15, 0.2)
wilcox.test(x)


    Wilcoxon signed rank test with continuity correction

data:  x
V = 9.5, p-value = 0.6845
alternative hypothesis: true location is not equal to 0

Warning message:
In wilcox.test.default(x) : cannot compute exact p-value with ties

According to this procedure, you have a tie because in the initial ranking step signs are ignored, so that $pm 0.2$ counts as a tie. But even with one tie, the P-value is not likely to be catastrophically wrong.

Difficulties with very small samples. You could get a significant result for a one-sided test (of $H_0: \eta =0$ vs. $H_a: \eta > 0$ with $n= 5$ if all five observations are either above or below $0.$ for example:

x = c(0.1, 0.3, 0.4, 0.5, 0.8)
wilcox.test(x, alt="gr")$p.val
[1] 0.03125

Power of Wilcoxon SR test with 15 observations. Here is a simulation with a million samples of size $n = 15$ from $\mathsf{Unif}(-.4, 1)$ rounded to one place, so that the population median is $\eta= 0.7.$ If we have such a sample, what is the probability that $H_0: \eta = 0$ will be rejected in favor of $H_a: \eta > 0\,?$

This probability is called the 'power' of the test. The simulation suggests that the power is about 90%. (There are warning messages; with 15 observations rounded to one place, there are almost sure to be ties, but we ignore error messages. A run with no rounding, hence no ties, gave almost the same result.)

set.seed(617)
pv =  replicate( 10^5, 
       wilcox.test(round(runif(15, -.4, 1),1), alt="g")$p.val )
There were 50 or more warnings (use warnings() to see the first 50)
mean(pv <= .05)
[1] 0.81138

The first of the one million runs gave the following result, of which we used only the P-value in the simulation:

set.seed(617)
x = round(runif(15, -.4, 1),1)
sort(x)
 [1] -0.4 -0.3 -0.1 -0.1  0.1  0.2  0.3  0.5  0.7
[10]  0.7  0.8  0.8  0.8  1.0  1.0

wilcox.test(x, alt="g")

        Wilcoxon signed rank test with continuity correction
data:  x
V = 103.5, p-value = 0.007189
alternative hypothesis: true location is greater than 0

Warning message:
In wilcox.test.default(x, alt = "g") :
  cannot compute exact p-value with ties

Summary. The one-sample Wilcoxon test seems best for data, with a couple of caveats: (a) If you can use samples larger than about $n = 10$ you will have a better chance of discovering when your observations are not centered at $0.$ (b) You might get fewer ties (thus potentially somewhat more reliable results) if you can use data with two (or more) decimal places rather than one.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your detailed answer. Luckily, my real data has many decimal places, so that should reduce the ties. Would those make a big impact? The Wilcoxon test certainly seems more conservative, but that may be a good thing. I just looked at an instance with only 3 observations - even if these are all negative and pretty far from 0 (-0.54, -0.23, -0.51), I'm getting p=0.25 with Wilcox. I get p=0.05 with a t-test in that case. However, with only 3 observations it seems fishy to ascribe significance anyway. $\endgroup$ – Scott Z Jun 18 '19 at 17:40
  • $\begingroup$ The distribution theory for Wilcoxon's rank-based tests becomes more complicated when ties are present. Most modern computer implementations, such as the one in R do a good job of giving accurate approximate P-values. So I don't expect huge differences by avoiding ties. For $n < 5,$ Wilcoxon SR test is essentially useless. (Even with $n=5,$ P-value of two-sided test can never be smaller than $1/16 > .05.)$ Reducing data to ranks loses information, so if data are normal Wilcoxon SR test cannot generally have a large a power as a t test, which is exact for normal data. $\endgroup$ – BruceET Jun 18 '19 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.