Analyzing repeated rank data. I have a data set of N people, T items. Let's say N=100, and T=10. 
Each person goes through the following exercise. 


*

*She is shown 2 random items from the set of T=10, and ranks them as rank 1 and 2. 

*She is next shown 2 more random items from the remaining 8 out of 10 items, and ranks as rank 1 and 2. 


At the end, the data set is of size 100x10, where each row has 4 numeric entries (two of which will be 1, and the other two will be 2) and 6 empty entries. 
My goal is to compare the 10 items against one another, and come up with an estimated rank value for a given item. 
What is the best way to analyze such data ? 
Thank you. 
 A: I think that using a rating system, like Elo or Glicko, is a good choice.


*

*Do the experiment with subject A, then repeat with subject B, subject C, and more.

*Randomize matches' (i.e. comparisons) order and insert results in a rating system engine.


If you're interested in use more than in development, rankade, our free ranking system (for sports, games, items, and more) is another option. It allows matches with both 2 and 3+ factions, while Elo and Glicko works just for one-on-one (here's a comparison). Due to your items, it should be easy - and useful - comparing 3 or 4 types in each test.
A: Here one possible answer, although I imagine a better one exists:
Take the row means (ignoring blanks).
A: This might be an odd approach but logistic regression might be useful. For example, if person 1 compared items 1vs2 and 3vs4 and person 2 compared 2vs3 and 5vs6, and if the lower # items were always rated as "2", the data could be entered in R as:
data.frame(
T1=c(1,2,3,4,2,3,5,6),
T2=c(2,1,4,3,3,2,6,5),
Y =c(1,2,1,2,1,2,1,2) 
)

All 3 variables are categorical. You can predict the probability of getting a "2" in Y from T2, controlling for T1. The p-values and standard errors will be meaningless because nesting was not taken into account but the estimates could be useful. 
Then, you can order the Items based on the probabilities that they will be ranked higher than another item.
