I have just read this paper about different test statistics one could use in Information Retrieval (IR), discussing whether Wilcoxon, Sign-test or Student t-test would be more reliable for comparing two IR systems.

According to the paper I read, the Sign and Wilcoxon tests do not take into consideration the statistic of interest (which is the difference in average precision); and by approximating them (both in different ways), we get a higher difference in p-value (generally higher), this leads to false positives and negatives.

In addition, the authors discourage using the Wilcoxon and Sign tests since they were built when computing randomized tests or Student’s t test was very expensive.

However, at the end of the paper, they mention that a small sample size (smaller than 50), "violations of normality may result in errors in the t-test."

So my questions come here: When we use Wilcoxon, Sign test, and T-test, which should we rely on? Are the points that I am trying to make valid?

  • 2
    $\begingroup$ I have attempted to edit this post for readability. To do so, I compared it to statements in the paper. Because this frequently caused the meaning to change, please check that the edits are consistent with your understanding. I wasn't able to find any reference to "very expensive," nor has it ever been the case that computations in Student's t test were difficult or time consuming, so your remarks about that are unclear. Finally, in reading through this post I don't see you making any points at all, so how are we to address your last question? $\endgroup$
    – whuber
    Apr 20, 2015 at 16:55
  • $\begingroup$ sorry I thought they meant that Wilcoxon was built because randomised test was too expensive to reproduce, they underline many times about how Wilcoxon is now old, since computation got better. I think my last question still holds. On which type of test should we rely more for such analysis? $\endgroup$
    – ramborambo
    Apr 20, 2015 at 17:26
  • $\begingroup$ If you mean to ask the question in your comment, you should edit your question ... but note that this then seems to ask much the same question. $\endgroup$
    – Glen_b
    Apr 25, 2015 at 19:56