What method is preferred, a bootstrapping test or a nonparametric rank-based test?

Question

I want to perform a single-tail test on a single sample of real numbers (N~100) against an expected value. The population is known to be not normally distributed. So from what I've read about stats, I can do my testing using

1. Wilcoxon signed rank test, or
2. bootstrap shifted sample data to obtain the null distribution of t-statistic (see How to perform a bootstrap test to compare the means of two samples?).

Is that correct?

Which method is preferred for minimizing type I error, and if possible, why please?

Answer 1

You just described the difference. No one can know in advance outcome differences because it greatly depends on the nature of your data.

Do you know the non-normal distribution you're working with? If so, you could simulate some results and see what the typical error rates for the different tests were and how they differed.

Answer 2

This answer may be helpful, and/or it may be annoying. Your welcome and my apologies at the same time :)

One thing to remember when using a normal distribution, is that it has a set of sufficient statistics, namely the mean and variance. What this indicates is that only the mean and variance matter in the inference. Any property of your sample besides the mean and variance will be thrown away when you use a normal distribution.

The statement that the "population is not normally distributed" is a bit of a misnomer - the population is not "distributed" at all - there is one and only one population (imaginary data sets and alternate worlds aside). It sounds like what you are actually saying is that your knowledge of the population consists of something other than the average and variance

So presumably, the only thing to do is to state what this extra/different knowledge is. Perhaps you know the skewness (or you know the skewness is important/relevant for the analysis, and not "noise").

I would suggest that you simply calculate the probability that your hypothesis is true, conditional on the information you have. This would include the data, and whatever "structure" you claim to know about the population that makes it non-normal (something other than the mean and variance of the population). So call your one sided test $T$, then you simply calculate:

$$P(T|D,I)=\frac{P(T|I)P(D|T,I)}{P(D|I)}$$

$P(T|I)$ is the prior probability for the test being "true" or "successful" (what did know about the test prior to seeing the data?). $P(D|T,I)$ is the "model" or "likelihood" and is similar to a p-value (How likely is the data you observed, given the test is true?). And $P(D|I)$ is often called the "evidence" (how well do any of the hypothesis predict the observed data?) - this quantity does not need to be explicitly assigned, as it can be derived from the requirement that the probability must add to 1.

The good thing about this method, is that probability theory will "construct the optimal test for you". You just need describe your prior information, and then simply do the mathematics. Now you may find that a bootstrap may be necessary in order to evaluate some mathematical formula - you may find that you should do the wilcoxon test - or probability theory will construct a test which is better than either of them (in terms of that type 1 and type 2 error you speak of).

Answer 3

The inferences generated by Wilcoxon vs bootstrapping cannot be compared as they pertain to different data. Wilcoxon is a rank test, thus generates inferences that pertain to ranks. Bootstrapping applies to the raw data, and thus generates inferences that pertain to the raw data. If you dislike bootstrapping but want inferences that pertain to the raw data, then you may want to try a permutation test (sometimes referred to as a randomization test).

Answer 4

I have two notes and one suggestion.

The first note is that testing theory is typically done by setting an acceptable level where you would reject a true hypothesis (Type I error), then minimize the risk of accepting a false hypothesis (Type II error). There are two reasons for this, first is that all your tests use this assumption, and second of all in almost all cases you can't minimize both errors simultaneously.

My second note is that the Wilcoxon Test hypothesis is actually $H_0: F_0 = F_1, H_1: F_0 \ne F_1$, where $F_i$ are CDFs, the relationship of this test to the mean is a property of the class of CDFs you are considering and the conditions you are considering them under.

Under the data discussed I think bootstrapping would probably be appropriate if you think the sample is representative of the population of interest. Other possible choices include deriving an empirical likelihood ratio test, or resampling t-tests and checking robustness.