All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a likert scale from 1-5 yields a greater amount of information.
Unless # of questions < # of potential answer per question.
But how much more?
I ask why because as I was pondering this question I realized whenever you provide a likert scale survey,
1) There's a tendency for the users to interpret the semantics of how you define the extremes or actually, most of the time, each number on the scale.
- "Most Epic Idea Ever Thought Of,"
- "2nd Coming of Christ"
"1" represents (
- "Not Interesting"
- "Not Really,"
- "I'd Rather Be Sprayed with Mace," etc.
Whereas if users rank 10 items in order from one extreme to the other, irregardless of however they interpret the extremes subjectively, the act of ranking limits their freedom to an ordered set. I realize this gets tricky with 5 questions vs. a 1-5 likert scale; I mention this only as a side note - not to distract from my main question.
If context matters... I've actually done this many times with people whom watch my Youtube channel. I want to know which video ideas/titles they would prefer to see produced next.
I brainstorm a list of 100 ideas or "titles."
I randomly arrange them as a list of 100 questions. Each user is to rate their preference for said video title on a 1-5 Likert scale.
I also record time taken and many, many other variables. Trust me, I consider all factors - however, in the end, with enough responses, the power rises above the noise and since the questions are randomized - even the noise of psychological fatigue, etc. drowns out.
I have not attempted asking them to rank 100 topics/ideas (yes, all 100 as a collection - it feels wrong but I am quite curious). Perhaps breaking them up into subsets would work, but their enthusiasm has already surprised me.
Irregardless of the context, assuming everything else is controlled how much more, or perhaps less information (bits/nats whatever is fine) would a ranking survey yield as opposed to a likert scale 1-5 one with 100 questions? What about 50 questions? Why?