# How to ascertain statistical significance for nonparametric tests if the sample size is small

In literature about nonparametric rank tests in event studies, I often encounter a variety of test statistics and authors often attempt to convince their readers that their methods overcome certain issues that others have or that they have other desirable properties. Tests in the event study space usually take a (financial) abnormal return and aim to uncover whether that return is significantly different from zero, given an estimation period.

One thing that I have noticed is that very often, a test statistic is derived and it is either implicitly assumed or explicitly stated that said test statistic follows a certain distribution, asymptotically.

For example, the authors of this paper derive the following test statistic for their nonparametric test:

$$t_{rank} = Z \sqrt{\frac{T-2}{T-1-Z^2}} \xrightarrow{d} t_{n-2}$$ as $$n \rightarrow \infty$$

This is merely an example and the details of the statistic are not of uttermost importance.

My question is: Considering how prevalent these asymptotic properties for nonparametric tests for event studies are, I wondered how one is able to test the statistical significance of a test statistic of a nonparametric test when sample size $$n$$ is small.

• The properties of some such tests can be derived analytically. Sometimes a normal approximation helps. My guess is most results are obtained by simulation. Apr 6, 2020 at 22:52

Following my Comment, here is an example of finding powers of two tests by means of simulation:

Suppose you have two samples of size $$n_1 = n_2 = 10$$ from normal distributions with $$\mu_1 = 1, \mu_2=3$$ and $$\sigma_1=\sigma_2 =1.$$

A two sample t test is the natural test to see if data are likely to show a significant difference between the means. What is the probability of 'detection'? This is the 'power' of the test. [Specifically, we want to know whether the test can use the two sample means to find a difference of 2 units (in either direction) between the two population means. We are willing to assume that the two groups have the same variance.]

But would it be a bad mistake to use the nonparametric Wilcoxon signed rank test instead? What is the power of this Wilcoxon to detect such a difference in locations?

For significance level 5%, a simple simulation in R shows the pooled 2-sample t test has probability almost 0.99 of correctly rejecting $$H_0$$. (The exact value 0.9882 can be found using the noncentral t distribution.)

set.seed(2020)
pv.t = replicate(10^5,
t.test(rnorm(10,1,1),rnorm(10,3,1),bvar.eq=T)$p.val) mean(pv.t < .05) [1] 0.98776 # aprx power for pooled t test  Similarly, simulation shows that the Wicoxon rank sum test (as implemented in R) has a slightly smaller probability of rejection, about 0.98. So the penalty for choosing the Wilcoxon test is not great. (An exact power for this nonparametric test would be more difficult to find analytically.) pv.w = replicate(10^5, wilcox.test(rnorm(10,1,1),rnorm(10,3,1),bvar.eq=T)$p.val)
mean(pv.w < .05)
[1] 0.98108      # aprx power for Wilcoxon SR test

• Thank you for your answer. I still struggle to see how that enables one to decide whether to reject/not reject a null. For instance, after calculating a test statistic (based on data from an unknown distribution), how is it possible to compute a $p$-value that allows to assess whether a hypothesis can actually be rejected or not, for a non-standard (i.e. one for which R does not provide a function with exact $p$-values built into it) nonparametric test?
– Kuma
Apr 7, 2020 at 8:25
• You have several related issues in your comment. Suggest you pick one at a time and ask fresh well-focused Questions for a couple of them. // Maybe like: "I gave the same test to 12 people; of whom a randomly chosen 6 took a traditional class and the remaining 6 studied the material online. The top five scores were from students who took the traditional class. Can I reject $H_0$ that methods of instruction were equally effective." // Maybe a somewhat different very specifically targeted Q of your own that will help you sort out what's unclear to you. // Anyhow best to take one issue at a time. Apr 7, 2020 at 16:32
• You're right, thanks for answering!
– Kuma
Apr 9, 2020 at 11:08