As part of my graduate thesis (area: psychology) I have gathered preference data. The data includes approximately 50000 heads-up comparison between elementX and elementX. I have a total of 15 elements. The participant would be shown 36 comparisons in random order, of which he/she would then choose the preferred element. I need to be able to calculate a meaningful "interval" scale-value for each element in order to see if the actual preferences of each element can predict theoretically generated scale values using regression analysis.
The Data: 14 elements, forming a 14x14 matrix table 91 unique possible comparisons ((14*14-14)/2) = 50000 comparison (NOT evenly distributed across all comparisons)
The problem: Since showing the participant all 91 possible comparison would be too time consuming I've just presented each participant 36 comparisons. Moreover, each participant was not eligible for each element, the eligible elements was determined if the participant owned the element.
For example: ParticipantX owns element; 1,5,6,8 and 10. And would therefore only be shown comparisons that includes those elements.
Furthermore, as owning some elements were considered more rare, I made a priority list for each element, so that element1 > element2 > .... > element14
For example: If the participant would be eligible for 50 possible comparisons (but restricted by the 36 comparison-cap) the comparisons shown to the participant would be based on the rarity order.
I'm afraid I destroyed the data by making this rarity order, since it resulted in some some elements being compared unequally to some elements; For example: of 1000 comparisons that element 1 was included in, 30% was with element3, 10% with element4 etc...
My questions: What method would you suggest would be the best for calculating "scale" values for each element? Currently I've calculated the basic probability that a given element has been choosen over another element. For example; element1 being chosen 95% of times to be preferred over any other element. However, I would prefer a more refined method to calculate the element values.
Thanks for your time, Chris, Finland.