I've got data from a psychology research study where participants are given scores for various different situations.
I'm looking at comparing the rank of those scores - is the person ranked as 1st in Situation 1 also ranked 1st in situation 2? Or is it inverted so that the person ranked 3rd out of 50 people is actually 47th, etc. That is a rather simplistic description but I'm looking at finding out comparing the rank structure between situations, and having some metric to describe how/if they differ. At the end, I was hoping on linking these shifts to psychological scores which I have measured for each participant (e.g. personality traits, etc)
I've thought about Wilcoxon signed-ranks tests but those involve mean ranks which is slightly different to what I"m looking to do.
I also thought of a rank correlation (e.g. Spearman's r). However, I am worried that the correlation would pick up on relatively small changes in rank out of a group of five hundred people, I would expect most people to shift up or down by a few ranks just through natural variation - I thought that this might get in the way of detecting those people who move up say by 50 or 100 ranks (i.e. into the next quartile). I also thought that the Spearman rank correlation would make it difficult to identify differences between those that shift upwards vs. those that shift downwards vs. those that do not shift at all.
Any other ideas?