How would you deal with lists of varying lengths? For example, if you asked for an ordered list of your favourite movies without any constraints how would you value the weight of someone who submits 5 samples and another who submits 20 samples?

Aside from the linear valuations, crudely you could give each participant's first selection a score of n, 2nd = n-1, 3rd = n-2... nth=1/nth value (n being the longest list supplied) and build an aggregated score across the sampled population. But that feels far too crude for proper analysis. Is there a better weighting distribution with a tapered decay to suit a long tail?

Instinctively, the person who supplies a longer submission should weigh higher than a shallow submission. But I'm unsure of the social research techniques that might be suitable.

Is there a statistical model that addresses this kind of analysis for open response lists?

  • $\begingroup$ I don understand why a respondent that listed and ranked 20 films should be more important than somebody listed and ranked 5. As for the scores for films, I'd compute them as 1/rank (or something like that). I.e. the most favourite receives 1, the second placed gets 0.5, the 3rd is 0.33. Differentiation between 11th and 12th intuitively must be much smaller than between 1st and 2nd. $\endgroup$ – ttnphns Apr 6 '13 at 21:09
  • $\begingroup$ Adrian, perhaps it would be easier for us to answer if there was a more general question you could start with. You've jumped right to how you would weight responses (presumably for developing a score to apply back to list items). But, I don't think we know yet if weighting is even the right approach (ttnphns's comment suggests something else entirely). $\endgroup$ – Jonathan Apr 8 '13 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.