# How to summarise the ordering pattern of a set of four stimuli by participants?

I did an experiment where I asked 13 participants to order 4 stimuli (let's call them stimulus A, B, C and D), so there are 24 possible ways to order 4 items.

Now I would like to know how to analyze this data in the correct way.

Indeed I would like to say that on average a certain order configuration has been preferred. For example I would like to say that considering all the configurations expressed by a participant, the following represents better the order preference of the participants

1. stimulus D
2. stimulus B
3. stimulus C
4. stimulus A
-
 You need to work on your profile, at least to come up with a better user name, and on your accept rate. – StasK Sep 21 '11 at 18:22 Paired-preference models have been discussed on these related threads: Testing the importance of an item among a finite set of items, Multinomial choice with binary observations. The problem here, as pointed out by @whuber in his comment, is the sample size. – chl♦ Sep 21 '11 at 19:02

You would want to convert the orderings into a categorical variable that takes 24 values, and analyze that variable. The test whether all combinations are equally likely will be given by a Pearson test with the null that all proportions are equal to 1/24. (By a rule of thumb of having an expected count of at least 5, you need at least 120 total participants.) For fancier analysis, you can formulate this as a multinomial logit model, again with 24 categories. (You would have to have all the possible combinations in the data set for this to work out.) An intercept-only model your model will tell you the relative probabilities (or rather log odds ratios against a selected baseline alternative), and you can test that all the coefficients (plus the baseline of zero) are smaller than the selected alternative. This is a difficult test per se, mind you, since you are talking about a multivarate one-sided alternative, so the distribution of the test statistic is a non-standard sum of $\chi^2$ with different degrees of freedom. (I am not sure you even want to mess with that; an extremely conservative approximation is to use $\chi^2_{23}$ or whatever the degrees of freedom of the test would come to be.)

Note that if you first pick the most frequently preferred combination in your data, and then test against, it would be data snooping with no control over type I error. The only way this test would work if your most preferred ordering came from some sort of substantive consideration (e.g., brandy is better than wine, which in turn is better than beer, which in turn is better than milk).

-
It appears this technique effectively ignores the information about preferences encoded by the 24 categories. Obviously it's not going to work with just 13 observations. But even in general it doesn't seem to address the question. – whuber Sep 21 '11 at 18:43
Ok, thanks to everybody for your precious answers and comments. The question now is: what I can say with only 13 participants? And If I had even less participants? Should I apply the techniques you suggested or maybe it does not make too much sense to have such a small sample? – Luca Sep 21 '11 at 23:06
13 isn't great sample but there's no reason to discard it. You could, for example, do what Greg recommended and obtain 13 rectangular matrices which you may cluster or multidimensionally scale. Also, Kemeny median could be computed, showing central tendency for your 13 rankings. – ttnphns Sep 22 '11 at 4:58
Oh. I was not paying attention. With 13 subjects, there's not much you can do, analytics-wise: there is not enough power for anything. MDS or some sort of profile analysis might help seeing the structure in the data, but that's about it. – StasK Sep 22 '11 at 18:15
Ok, thanks a lot to everybody for the precious help. – Luca Sep 24 '11 at 14:09

One approach would be to think in terms of pairwise comparisons and apply the Bradley Terry model. A person who ranks the items as D,B,C,A would be equivalent of saying D beats B, D beats C, D beats, A, B beats C, B beats A, and C beats A. Then another person with a different order would result in a different set of pairwise comparisons. You would then combine all the pairwise comparisons and get a set of rankings of the 4 choices.

-
Let me add to Greg Snow's answer that, doing pairwise comparisons, you ought probably to recognize psychological exponentiality of ranking. Usually, drop from 1st place to 2nd is steeper than from 2nd to 3rd, and so on. So, in ranking D,B,C,A you might let distance D-B be 1, B-C be, say, 0.5, C-A be 0.25. – ttnphns Sep 21 '11 at 19:23
Yes, I see the point. Actually the distance between stimuli in my case is identical. – Luca Sep 21 '11 at 23:11