best approach to compare ranks over time

Question

If I have the following dataset:

rankdata <- read.table( text = " time  id  rank
                             1     a    1
                             1     b    2
                             2     a    2
                             2     b    3
                             2     c    1
                             3     a    1
                             3     b    3
                             3     c    2
                             4     a    2                     
                             4     c    1
                             5     a    1                                 
                             5     c    1 ", header = TRUE)

I am trying to think of the best way to test whether ranks are consistant over time or not. I obviously work in R so pointers to doing this there would be best.

Absent ids within time points are not unmeasured data points but ids that were not present at the time of ranking. So number of ranks within a time point may be 1-6 then 1-12 in the next but generally within this range. Also note I expect that there may be ties.

What if I had several data sets by say island, I would need to incorporate island as a random effect. Each island will consist of completely independent ids.

Answer 1

Here is one way to try. Construct a time series for each comparison of "X ranks better than Y"; with the values of win, lose, tie or NA. So for from your example data you have:

a-b: win  win  win  NA   NA
a-c: NA   lose win  lose tie
b-c: NA   lose lose NA   NA

Then you need to decide for yourself what you mean by consistent over time - what tolerance margin of inconsistency do you have? For example, the a-c comparison already shows inconsistency. Is this enough to throw it out?

You will also need to decide if you therefore want to reject the consistency hypothesis just for the a-c combination; for the whole system; or perhaps just for a and c (and there are various ways you might give yourself consistency measures for the margins ie all a, all c, etc).

If you want to allow a certain tolerance for minor inconsistency you will need to create some underlying latent random variables with distributions you can specify. This becomes your null hypothesis and you use it to generate simulated data that will give you an idea of the tolerable inconsistency; which you can compare somehow (with whatever summary statistic/s seems most relevant) to what you have actually observed.

On the island question, as the ids being ranked are not in common across islands I think you can treat each island as a completely independent problem.