Why is the null hypothesis for Wilcoxon test not rejected if the sample size is 5? If I perform a simulation for the one-sample Wilcoxon test (2-sided) for samples of size 5 (1000 times), the null hypothesis could not be rejected. Why?
I know the probability of the test statistic being equal to zero is 1/32= 0.03, which less than $\alpha=0.05$. However, I do not get p-values less than $\alpha$.
 A: I'll write "differences" when describing the observed values (as if it were a paired test), but for a one-sample test just read 'difference' as 'observation' or 'value'.
The possible values of the test statistic (here using Wilcoxon's original definition of the statistic) for the sum of the ranks for the negative differences are $\{0, 1, ..., 15\}$  ($0$ being the value when all five differences are positive, and $15$ being the value when all five differences are negative).
These have probability $\frac{1}{32}$ times $(1,1,1,2,2,3,3,3,...,1)$ (the probabilities are symmetric around the mean sum of ranks of $\frac{15}{2}$).
The smallest possible two tailed p-value is therefore $\frac{1}{32}+\frac{1}{32} = \frac{1}{16}>0.05$. You cannot see a smaller p value than this.
Consequently, you cannot reject $H_0$ at the $5\%$ level with $n=5$; the smallest available two-tailed significance level is $\frac{1}{16}$.

This is a common problem with extremely small samples and various commonly used discrete test statistics (e.g. those used in sign tests, Fisher exact tests, rank correlation tests, the usual rank tests of location, etc). I have on multiple occasions seen researchers conduct tests at small sample sizes in these circumstances without being aware that there was no possible sample that could lead them to reject $H_0$.
If significance levels are lower, such as when there are corrections for multiple testing, this problem will occur more often.
This is one reason (of several) why I encourage people to explicitly choose a rejection rule based on a test statistic, not simply to compare $p$ to some $\alpha$ (a potentially dangerous practice, as we see here). Explicitly identifying a rejection rule at least increases the chance that the problem (that no such rule exists at the desired significance level) will be recognized.
Indeed, when the statistic is discrete I encourage people to identify the set of available significance levels (or at least the ones near desired significance levels), and to choose from among them.
There are sometimes ways to mitigate the discreteness issue, at least with some statistics, without altering the interpretation of what the statistic measures -- and without losing the distribution-free property of permutation statistics when performing such tests. Such approaches can't help in this case, though.
