Mann–Whitney U test shows there is a difference between two sample sets, how do I know which sample set is better? I have two algorithms which output a performance number rating their performance each time they are executed. Higher is better. So now I have two sample sets of data, $S_1$ and $S_2$.  $S_i$ contains performance numbers from algorithm $i$, $i=1,2$.
If I use the Mann–Whitney U test to reject the null hypothesis and determine that the distribution of $S_1$ and $S_2$ are not equal, I have determined that one of these algorithms produces better performance than the other. How do I know which algorithm is producing the better performance? Is it as simple as  determining which sample set, $S_1$ or $S_2$, has the higher mean value?
 A: The Mann-Whitney U test allows for one-sided hypotheses, e.g. $S_1$ is stochastically greater than $S_2$. If computing the test statistic with $S_1$ ranks, U will bottom out at $0$ (the case where all observations in $S_1$ are less than those in $S_2$) and be maximal at $|S_1||S_2|$ (the case where all observations in $S_1$ are greater than those in $S_2$). As our test statistic $U$ tends towards either of these extremes, we can make one-sided inference.
So, to the question of which algorithm performs better, you may perform two one sided tests (both the alternatives that $S1$ is greater than $S2$ and that $S_1$ is less than $S_2$), apply some correction to the test results (e.g. a Bonferroni correction would effectively double each p-value), and then select an alternative with a significant corrected test result. At most one alternative test may be significant, and when neither test shows significance, we cannot infer that one algorithm is better than the other.
Most computational tools should provide easy access to one-sided tests. For example in R, the 'wilcox.test' function accepts an 'alternative' argument which can specify the alternative. There's even an online calculator which provides a one-sided test.
