Comparing ranks within groups

Question

I've got data from a psychology research study where participants are given scores for various different situations.

I'm looking at comparing the rank of those scores - is the person ranked as 1st in Situation 1 also ranked 1st in situation 2? Or is it inverted so that the person ranked 3rd out of 50 people is actually 47th, etc. That is a rather simplistic description but I'm looking at finding out comparing the rank structure between situations, and having some metric to describe how/if they differ. At the end, I was hoping on linking these shifts to psychological scores which I have measured for each participant (e.g. personality traits, etc)

I've thought about Wilcoxon signed-ranks tests but those involve mean ranks which is slightly different to what I"m looking to do.

I also thought of a rank correlation (e.g. Spearman's r). However, I am worried that the correlation would pick up on relatively small changes in rank out of a group of five hundred people, I would expect most people to shift up or down by a few ranks just through natural variation - I thought that this might get in the way of detecting those people who move up say by 50 or 100 ranks (i.e. into the next quartile). I also thought that the Spearman rank correlation would make it difficult to identify differences between those that shift upwards vs. those that shift downwards vs. those that do not shift at all.

Any other ideas?

Simon.

Answer 1

There is a long history of the use of rank statistics in statistical analysis. Since the 1990's, there has been somewhat of a unification in the field as well as the development of underpinning theory in what was once partly statistical folklore.

The following reference provides a basic review of the ideas and concepts:

MG Akritas and E Brunner (2003) Nonparametric models for ANOVA and ANCOVA: A Review. In Recent Advances and Trends in Nonparametric Statistics. MG Akritas and DN Politis, Editors. Elsevier Science B.V.

For more applications in the field of plant pathology, see:

D. A. Shah and L. V. Madden (2004) Nonparametric analysis of ordinal data in designed factorial experiments Phytopathology 2004 94:1, 33-43

I have used these methods with plant disease rating data, though that was several years ago. At that time, there were only SAS macros available which needed a little effort to use properly. However, now there are several R packages that implement these methods. In particular the nparLD package looks pretty useful:

N Kimihiro, YR Gel, E Brunner, and F Konietschke (2012) nparLD: An R software package for the nonparametric analysis of longitudinal data in factorial experiments. Journal of Statistical Software September 2012, Volume 50, Issue 12.

For your situation, it sounds as though you have repeated measures and several covariates. One way to work with those data is to stratify into groups based on the covariates. It looks as though the analogy to ANCOVA is not as nicely implemented yet. The following reference may be useful:

Haritini Tsangari and Michael G. Akritas. 2004. Nonparametric ANCOVA with two and three covariates. J. Multivar. Anal. 88, 2 (February 2004), 298-319.

The big upside to using this framework is that you have the ability to better model the actual structure of your study, since repeated measures and factorial design structures are both supported.