1
$\begingroup$

Let us say that we have 5 sets of data: $$ a = (a_1, a_2, \dots, a_{10}) \\ b = (b_1, b_2, \dots, b_{10}) \\ c = (c_1, c_2, \dots, c_{10}) \\ d = (d_1, d_2, \dots, d_{10}) \\ e = (e_1, e_2, \dots, e_{10}) $$

Let us suppose that we want to find out if some data are bigger than others. We perform for example the Wilcoxon signed rank test among each pair and end with a 5x5 matrix of p-values.

This is a lot of data and the relationships are not immediately clear. Moreover if I have many instances of these data, then I need a new matrix for each instance. Presenting data like this for 40 instances is impossible in a short conference paper. I am looking for some methods on how to use some algebraic properties of significance tests to make this data more readable and less bloated.

One way would be to:

  1. Identify all those sets of data for whom there is no other data which is significantly better
  2. Mark these sets data as "first grade"
  3. Identify all those sets of data for whom there is no other data, except for "first grade" data, which is significantly better
  4. Mark these sets of data as "second grade"
  5. continue to put all data into grades

Then I would just write to which grade does a data set belong. Like that I can fit all relevant information into a 5x40 table.

So far I found just one problem with this. That is the Wilcoxon test as well as all ranking tests are not transitive. That is I can get a situation where for each data set there is some other data set which is significantly better and in this case my procedure would fail.

Is there some standard, beautiful way to present data such as this?

$\endgroup$

1 Answer 1

0
$\begingroup$

You could make a directed graph with arrows indicating what's significantly better than what. That shows every relationship without forcing an ordering. However, you could still say, whoever gets most outgoing arrows is probably the best.

Also, it you get a "rock-paper-scissors" situation as discussed in the blog post you link to, then I would say your criterion for ranking is not very useful for interpreting the results.

In the example of the dice where each die is likely to beat the next in a circular rock-paper-scissors way, you need to figure out what's important: it is important to maximize the probability of getting the highest of all dice? is it important to minimize the probability of having the lowest? to optimize the overall average ranking? these criteria should produce non-equivocal rankings. Of course, it doesn't mean it's easy to figure out where there is statistical significance once you change the criterion...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.