Testing the importance of an item among a finite set of items

Question

Consider the following survey question:

Q: How would you classify the importance for you of the following 5 items:

A
B
C
D
E

Assign to each item a number in the set {1,2,3,4,5}, with 1 meaning the highest importance and 5 meaning the lowest importance; the number used to an item cannot be used in any other item.

• How can one test whether item A is the most important item for a sample with 100 individuals?

Answer 1

The naive approach would be to compute the marginal distribution of rankings (e.g., mean score for each item), but it would throw away a lot of information as it does not account for the within-person relationship between ranks.

As an extension to paired preference model (e.g., the Bradley-Terry model, described in Agresti's CDA pp. 436-439), there exist model for ordinal or likert-type comparison data with or without subject covariates, as well as model for ranking data (baiscally, it relies on the use of log-linear model). Here is a short intro to the package, and a mathematical explanation in this technical report: Fitting Paired Comparison Models in R. You will find everything you need in the prefmod R package, see the pattR.fit() function which expects data in the form you described:

 The responses have to be coded as consecutive integers starting
 with 1.  The value of 1 means highest rank according to the
 underlying scale.  Each column in the data file corresponds to one
 of the ranked objects. For example, if we have 3 objects denoted
 by ‘A’,‘B’,and ‘C’, with corresponding columns in the data matrix,
 the response pattern ‘(3,1,2)’ represents: object ‘B’ ranked
 highest, ‘C’ ranked second, and ‘A’ ranked lowest. Missing values
 are coded as ‘NA’, ties are not allowed (in that case use
 ‘pattL.fit’.  Rows with less than 2 ranked objects are removed
 from the fit and a message is printed.

For additional information (about and beyond your particular study), you might find useful the following papers:

1. Böckenholt, U. and Dillon, W.R. (1997). Modelling within-subject dependencies in ordinal paired comparison data. Psychometrika, 62, p.411-434
2. Dittrich, R., Francis, B., Hatzinger, R., and Katzenbeisser, W. (2006). Modelling dependency in multivariate paired comparisons: A log-linear approach. Mathematical Social Sciences, 52, 197-209.
3. Maydeu-Olivares, A. (2004). Thurstone's Case V model: A structural equations modeling perspective. In K.van Montfort et al. (eds), Recent Developments on Structural Equation Models, 41-67.