2
$\begingroup$

Imagine I organized a race between 2 types of ants. Type A and B. At the end of the race, I have the ranking, which looks something like :

1 A1
2 A3
3 B5
4 A4
5 B2
...

Where, in A1, A is the ant type and "1" the ant ID. Ants do not interfere between each other during the race.

If I plot the proportion of A or B every 10% of the ranking it gives this (fake data):

ants

Maybe bar plot is better here. But one needs to read it as follow :

There are 1000 ants in the race, I arbitrary divide the ranking in 10. When

  • N = 1, that is the first 100, there are 54 Bs and 46 As.
  • N = 2, there are 50 As and Bs
  • N = 3, 55 As and 45 Bs
  • N = 4, 61 As and 39 Bs
  • etc..

My question is :

I have just ranking results and I want to known whether there is some bias in the ranking as a function of the type. eg. For a given ant, can one predict where it will rank ?

For instance, how can we say whether A-ants are very good at arriving in first positions 10%), and B-ants are very good at arriving in the 30-50% of the ranking, or not, but both are equally good at performing bad. (I don't want to test for this, but how can I end up with results of that kind).

I have many replicate of this experiment.

I don't really know how to test for this. I was thinking of:

  • dividing the ranking in N parts (how to choose N ?)
  • Doing a $\chi^{2}$ test for each part, where I expect 50% of type A and B under null hypothesis. : No because parts are dependants.
  • I don't know.

How would you do, and why ?

Note that the ranking doesn't include values (eg. time), so we only have ranking information.

In case we would have the time value, how would you do, if different ?

I would appreciate some re-wording of the problem, since I'm not a statistician and it prevents me to search in books (because I don't known where to search)

Thanks

$\endgroup$

2 Answers 2

1
$\begingroup$

You could do a two-sample t test on the ranks to check the null hypothesis of equal average rank. This is basically the same as a Wilcoxon rank-sum test on the original times for the null hypothesis 'no difference between A and B in typical performance'.

The analysis will be easier if you split rank and type in two columns (one with the rank and/or time and one with the type).

$\endgroup$
3
  • $\begingroup$ But doing a t-test on ranks won't tell me if there is a difference in a certain portion of the ranking, like in the first 10%, do we have more A-type or B-type ? $\endgroup$
    – jrjc
    Jul 16, 2014 at 14:29
  • $\begingroup$ Assuming equally shaped distributions of ranks between A and B will allow you to conclude such things. $\endgroup$
    – Michael M
    Jul 16, 2014 at 14:35
  • $\begingroup$ But I think that I actually want to test for equally shaped distribution, don't I? Sorry if I'm not clear, I edited the question. Could explain a bit more then ? $\endgroup$
    – jrjc
    Jul 16, 2014 at 14:42
1
$\begingroup$

I think there are some issues with the question as posted by @jeanrjc. I will address some of the issues and try to answer some of them, but I do not know the whole answer.

First, the figure in the question does not seem to be what the OP said it is. It does not seem to be the proportion of A and B IN the Nth ranking (in the 20% ranking) but seems to be the number of As and Ba UP TO the Nth ranking. This will explain why at the end of the figure the proportions of As and Bs stabilizes around 50% - 50% of the ants that terminated up to the 100% ranking are A and 50% are B!

Second point: the OP seems to be interested in performing both an exploratory analysis and a confirmatory analysis with the same data! For example, one can see that at 40% the differences between A and B seems to be at the highest, and so the p-value of any test would be at the lowest. One can make then a very high evidence statement that B ants are much more likely to arrive in the top 40% than A ants. What happened is that one looked at the data to find the maximal difference between A and B (the exploratory part) and then performed a test to verify that the difference was significant (the confirmatory part). Another example is the OPs statement: "but both are equally good at performing bad." - just select a region where there is the least difference between A and B, and then show that the differences are not significant in this region.

This mixing of exploratory and confirmatory analysis ON THE SAME DATA makes me very uncomfortable (I dont know the name of the this "error" in statistics, but in machine learning it is sometimes called overfitting - the idea that one can select the best parameters/statements on a "test" set and expect that it will work as well on yet unknown data). If one must do both a exploratory and a confirmatory analysis, I would do it an different sets of races. Select a random set of races to find the "best statements" and test these statements on the remaining races.

The third issue is that a hypothesis testing require a hypothesis to be tested. Isn't there statements that are meaning full to your problem that you can define before looking to the data? If you want to find out if A ants are faster than B ants, is not enough to test if A ants are significantly more likely to be among the top 50% than B ants?

Maybe the OP does not have a single statement but wants to test 10 statements of the form: A ants are more likely then B ants to be among the top N%, for N=10, 20, 30..100 In this case there is the fourth problem: multiple comparisons. If you make 10 statements/tests, each with a 5% change of being wrong (95% confidence on your tests) then there is a 50% chance that one of your statements will be wrong ( well not exactly but close). Thus one must be careful on how many tests are being made and adjust the confidence level on each test appropriately. Please follow the tags: , or in CV.

The last issue is that if one decides to make multiple tests, these tests are not independent. For example, given that the difference between A and B are at the highest/most significant at the 40% according to the figure in the question, it is unlikely that the difference will not be significant at the 50%. In other words, given that you already know that B is more likely than A to be among the top 40%, I dont think the correct null hypothesis for the 50% is that they are both equally likely! But I dont know how to solve this problem.

$\endgroup$
11
  • $\begingroup$ Thanks. But in the figure, it is the proportion in the Nth ranking (there are 100 ants per N, so 46 and 54 when N=1, and 50 ants of each type for N=2). It's actually fake data here, but still. Otherwise I totally agree with you on the issues. But I don't want to perform confirmatory analysis, I gave an example of what kind of result I expect, but I do not assume them in the first place. Basically I have just ranking results and I want to known whether there is bias in the ranking as a function of the type. eg. Given ant A3, can one predict where it will rank ? $\endgroup$
    – jrjc
    Jul 21, 2014 at 8:20
  • $\begingroup$ Thanks for taking the time and making the comments that I also had in mind. What you call "overfitting" is sometimes coined by the term "data driven". If done in such excessive way as in the comments by jeanrjc, the probability of having a type 1 error will be much, much higher than the chosen significance level. That's why I stopped defending my simple answer. $\endgroup$
    – Michael M
    Jul 21, 2014 at 8:50
  • $\begingroup$ @MichaelMayer : Can you explain more please ? What is excessive ? Why don't you want to detail your answer ? $\endgroup$
    – jrjc
    Jul 21, 2014 at 9:40
  • $\begingroup$ @jeanrc if indeed your graph plot the proportion of ant types that arrived between the N-1 and the N ranking, then I believe my last point is less relevant - but there is still some dependency between each test - the proportion of ants A that arrive between the 40th and the 50th position is dependent on how many of the A ants have already arrived, thus the null hypothesis should not be that the proportion of each ant is 50%. $\endgroup$ Jul 22, 2014 at 22:17
  • $\begingroup$ @JacquesWainer True, Chi-squared is also a bad idea for this reason, right ? How would you do then ? $\endgroup$
    – jrjc
    Jul 23, 2014 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.