# best approach to compare ranks over time

If I have the following dataset:

rankdata <- read.table( text = " time  id  rank
1     a    1
1     b    2
2     a    2
2     b    3
2     c    1
3     a    1
3     b    3
3     c    2
4     a    2
4     c    1
5     a    1
5     c    1 ", header = TRUE)


I am trying to think of the best way to test whether ranks are consistant over time or not. I obviously work in R so pointers to doing this there would be best.

Absent ids within time points are not unmeasured data points but ids that were not present at the time of ranking. So number of ranks within a time point may be 1-6 then 1-12 in the next but generally within this range. Also note I expect that there may be ties.

What if I had several data sets by say island, I would need to incorporate island as a random effect. Each island will consist of completely independent ids.

-
One of the awkward features looks to be that not all ids are ranked in each time-specific ranking exercise, is that correct? Also, in your extension to several island data sets, are the same things (a, b and c) being ranked on each island? – Peter Ellis Dec 16 '12 at 0:41
yes, however, the absent ids within time points are not unmeasured data points but ids that were not present at the time of ranking. Each island will consist of completely independent ids. PErhaps I willadd this to the question. – user1322296 Dec 16 '12 at 0:45

Here is one way to try. Construct a time series for each comparison of "X ranks better than Y"; with the values of win, lose, tie or NA. So for from your example data you have:

a-b: win  win  win  NA   NA
a-c: NA   lose win  lose tie
b-c: NA   lose lose NA   NA
`

Then you need to decide for yourself what you mean by consistent over time - what tolerance margin of inconsistency do you have? For example, the a-c comparison already shows inconsistency. Is this enough to throw it out?

You will also need to decide if you therefore want to reject the consistency hypothesis just for the a-c combination; for the whole system; or perhaps just for a and c (and there are various ways you might give yourself consistency measures for the margins ie all a, all c, etc).

If you want to allow a certain tolerance for minor inconsistency you will need to create some underlying latent random variables with distributions you can specify. This becomes your null hypothesis and you use it to generate simulated data that will give you an idea of the tolerable inconsistency; which you can compare somehow (with whatever summary statistic/s seems most relevant) to what you have actually observed.

On the island question, as the ids being ranked are not in common across islands I think you can treat each island as a completely independent problem.

-
hmm, this seems to frame things to me like binary matrix mantel tests across time points? – user1322296 Dec 16 '12 at 1:23
I hadn't thought of it quite in those terms, but that might be one way of comparing the matrix at time 1 with a matrix at another time. I'm not sure it would work with all the NAs. Basically, you do need to create a way of quantifying the inconsistency, unless you have zero tolerance. – Peter Ellis Dec 16 '12 at 1:36