I am trying to pick one from these two tests to analyze paired data. Does anyone know any rules of thumb about which one to pick in general?
$\begingroup$
$\endgroup$
7
-
$\begingroup$ Silverfish's answer there just (barely) touches on it. That question is pretty general, I wonder if we might be able to tolerate a more specific one. $\endgroup$– Glen_bDec 7, 2015 at 3:29
-
$\begingroup$ Sheldon -- the signed rank test carries an assumption about symmetry of differences that the sign test does not. On the other hand, if there is near-symmetry and the tail is not very heavy, the signed rank should have more power. $\endgroup$– Glen_bDec 7, 2015 at 3:31
-
$\begingroup$ I agree. In my case, the rank sum test has the largest p-value, sign test is the medium, signed-rank is the smallest. Therefore, it has more power. $\endgroup$– SheldonDec 7, 2015 at 3:35
-
$\begingroup$ @Sheldon No, that's not how you decide a test has more power - a lower p-value in respect of one sample may simply be due to the vagaries of that sample, whereas power is about the behavior across all random samples drawn from the same population. I had better write an answer that expands on the previous comments, explains what it means to have more power and explains some of the circumstances under which each might do better. $\endgroup$– Glen_bDec 7, 2015 at 4:12
-
4$\begingroup$ @Glen_b, I would say that currently the most important consideration is what will be most helpful for future readers. I think whoever searches for sign test vs Wilcoxon test and finds this thread, will benefit a lot more from reading your specific answer here than from being redirected to that mega-thread where they are likely to get lost and never find any answer. $\endgroup$– amoebaDec 7, 2015 at 23:02
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
2
I am trying to pick one from these two tests to analyze paired data. Does anyone know any rules of thumb about which one to pick in general?
The signed rank test carries an assumption about symmetry of differences under the null that the sign test need not. (That assumption is necessary in order that the permutations of the signs attached to the unsigned ranks of differences be equally likely.)
On the other hand, if there is near-symmetry in the population and the tail is not very heavy, the signed rank should have more power.
[This should not be taken as advice to choose between them on the basis of the sample; in general that leads to test properties different from the nominal (tests may be biased, actual significance levels are no longer what they appear to be, calculated p-values don't represent true p-values and so on). Instead, where possible, characteristics should be evaluated based on knowledge external to the sample the test is applied to -- whether by subject area knowledge, familiarity with other data sets like this one, sample-splitting, ...]
In my case, the rank sum test has the largest p-value, sign test is the medium, signed-rank is the smallest. Therefore, it has more power.
That's not how you decide a test has more power - a lower p-value in respect of one sample may simply be due to the vagaries of that sample, whereas power is about the behavior across all random samples drawn from the same population.
That is, imagine that you're dealing with some specific situation in which the population of pair-differences are centered somewhat away from 0 (i.e. that $H_0$ is false in a specific way). Then under repeated sampling under the same conditions (including sample size), the power will be the rejection rate for that particular population.
In similar fashion we could calculate the rejection rate for a sequence of populations with different location* of pair-differences and obtain an entire power-curve. Then "higher power" would correspond to the entire power curve (or almost all of it, noting that both should be at the same significance level) for one test laying above the other.
* you could take it to be a median for the present discussion -- while the estimator for the signed rank test is the median of pairwise averages of pair-differences, under the symmetry assumption the location estimator should also be a suitable estimate of median pair difference.
Here's a related question How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples. One of the answers includes a (brief) discussion of the present issue.
-
$\begingroup$ Thanks for your clarification. I think the most important take-home message is the assumption about symmetry of differences for signed-rank test, which is violated in my case. I have a feeling that other than checking whether the symmetry criteria is satisfied, there is no way to tell which test is wrong. Rather, it sounds reasonable to say which one is more appropriate. $\endgroup$– SheldonDec 7, 2015 at 15:46
-
1$\begingroup$ @Sheldon If you're not confident that the conditions for the test are close to true, you should generally not assume they are. Which is to say, perhaps the sign test would be a better idea. I hope to add some further information to my answer when I have a chance. $\endgroup$– Glen_bDec 9, 2015 at 0:15