Finding direction of inequality for two-tailed test

Question

I'm comparing two sets of measurements using a paired difference test (specifically the Wilcoxon signed-rank test). I would like to find out whether there is a significant difference between these sets of measurements, and also the direction of this inequality.

Since I do not have a theoretical reason to suspect this direction before I look at the data, I'm using a two-tailed test. This means my null hypothesis is that both sets of measurements belong to the same distribution; the alternative hypothesis is that the two measurements belong to different distributions.

The test has a sufficiently small p-value for me to reject the null hypothesis. From this, I conclude that there is a difference between these measurements. To find out the direction of the inequality, I looked at the mean of both samples. My supervisor commented that this was strange, since the Wilcoxon test evaluates rank as opposed to mean. He suggested that I use a one-tailed test instead.

My question is: is it appropriate to use a one-tailed test to find out the direction of the inequality? Or should I simply use a different aggregation technique (instead of the mean) to compare my distributions? The reason I'm hesitant is because a one-tailed test is normally used when I know the direction of the inequality a priori, which is not the case here.

Answer 1

If you want to find the direction of difference in the sample that led to rejection, you could simply compare the mean ranks (since that's what the test effectively compares).

If you're looking for an estimate of location difference, you want the Hodges-Lehmann estimate, and indeed there's a CI you can get for the location difference as well. Some packages will calculate it for you. [However, the Wilcoxon signed rank test can be sensitive to shifts in distribution other than a pure location shift.]

Here's an example in R (these data come from the first example in the help on the wilcox.test function in R):

   x <- c(1.83,  0.50,  1.62,  2.48, 1.68, 1.88, 1.55, 3.06, 1.30)
   y <- c(0.878, 0.647, 0.598, 2.05, 1.06, 1.29, 1.06, 3.14, 1.29)

   wilcox.test(x, y, paired = TRUE, conf.int=TRUE)

        Wilcoxon signed rank test

data:  x and y
V = 40, p-value = 0.03906
alternative hypothesis: true location shift is not equal to 0
95 percent confidence interval:
 0.010 0.786
sample estimates:
(pseudo)median 
          0.46 

So with a p=value of 0.039, we'd reject at the 5% level (two-tailed). The pseudomedian is 0.46, indicating that the tendency of x-y to be above 0.

Here's a plot of the pairwise differences, with the pseudomedian and the interval for the shift drawn in.

$\quad$

As you see, the low end of the interval sits just above the $0$ line (as it should if the CI corresponds to the rejection in the test).

If you assume that the distribution of differences is symmetric (as you would need to hold under the null for the signed rank test anyway), this sample pseudomedian is also an estimate of the population median of the differences (and the population mean of differences, for that matter, if the population mean exists).