The Wilcoxon signed-rank test without symmetry caused by one outlier

Question

I am comparing two algorithms on the same input data. Now I want to see whether the difference in output is significant. For this I need to use the Wilcoxon signed-rank test, since my data is paired and not normally distributed. However, there is one big outlier in the differences of the algorithms, which causes it to be non symmetrical (skewed). My question is, how should I proceed? I could transform the data such that this outlier has less of an effect. However, I'm unsure whether running the Wilcoxon signed-rank test on transformed data can still lead to a meaningful conclusion for the untransformed data. I could also do a paired sign test, which does not assume the differences to be symmetrical, but this test has less statistical power. The final option is to remove the outlier to regain symmetry, but it seems like a bad practice to me to remove an observation that causes an outlier.

Answer 1

As the name suggests, the Wilcoxon signed-rank test is a non-parametric based on ranks (ie, it analyzes ranks of differences, not the differences themselves) and ranks are not sensitive to outliers (the ranks of n numbers range from 1 to n, whether there are outliers or not).

Let's show this with an example.

differences <- c(
  -0.14, -0.08, 0.01, 0.43, 0.49, 0.59, 0.62, 0.95, 1.02
)
differences_with_outlier <- c(
  -0.14, -0.08, 0.01, 0.43, 0.49, 0.59, 0.62, 0.95, 100.02
)

Once we sort the differences, the biggest positive difference has the highest rank; the magnitude of the difference is irrelevant.

rank(differences)
#> [1] 1 2 3 4 5 6 7 8 9
rank(differences_with_outlier)
#> [1] 1 2 3 4 5 6 7 8 9

This property of ranks makes rank-based tests robust to outliers. Note the p-value, and therefore the conclusion from the signed-rank test, is the same with and without the outlier.

wilcox.test(differences)
#> 
#>  Wilcoxon signed rank exact test
#> 
#> data:  differences
#> V = 40, p-value = 0.03906
#> alternative hypothesis: true location is not equal to 0

wilcox.test(differences_with_outlier)
#> 
#>  Wilcoxon signed rank exact test
#> 
#> data:  differences_with_outlier
#> V = 40, p-value = 0.03906
#> alternative hypothesis: true location is not equal to 0