All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a likert scale from 1-5 yields a greater amount of information.

Unless # of questions < # of potential answer per question.

But how much more?
And why?

I ask why because as I was pondering this question I realized whenever you provide a likert scale survey,

1) There's a tendency for the users to interpret the semantics of how you define the extremes or actually, most of the time, each number on the scale.

"5" represents

  • "Best,"
  • "Definitely,"
  • "Love,"
  • "Most Epic Idea Ever Thought Of,"
  • "2nd Coming of Christ"


"1" represents (

  • "Not Interesting"
  • "Dislike,"
  • "Not Really,"
  • "NO!,"
  • "I'd Rather Be Sprayed with Mace," etc.

Whereas if users rank 10 items in order from one extreme to the other, irregardless of however they interpret the extremes subjectively, the act of ranking limits their freedom to an ordered set. I realize this gets tricky with 5 questions vs. a 1-5 likert scale; I mention this only as a side note - not to distract from my main question.

If context matters... I've actually done this many times with people whom watch my Youtube channel. I want to know which video ideas/titles they would prefer to see produced next.

I brainstorm a list of 100 ideas or "titles."

I randomly arrange them as a list of 100 questions. Each user is to rate their preference for said video title on a 1-5 Likert scale.

I also record time taken and many, many other variables. Trust me, I consider all factors - however, in the end, with enough responses, the power rises above the noise and since the questions are randomized - even the noise of psychological fatigue, etc. drowns out.

I have not attempted asking them to rank 100 topics/ideas (yes, all 100 as a collection - it feels wrong but I am quite curious). Perhaps breaking them up into subsets would work, but their enthusiasm has already surprised me.

Irregardless of the context, assuming everything else is controlled how much more, or perhaps less information (bits/nats whatever is fine) would a ranking survey yield as opposed to a likert scale 1-5 one with 100 questions? What about 50 questions? Why?

  • 1
    $\begingroup$ For me, this is easy to answer. I try to do my best on Likert questions, but always skip ranking questions. On what basis do I rank any two of: Ending climate change, Ending the Syrian civil war, Funding a Mars mission, Improving airports, Reducing import duties on coffee, Banning telephone solicitation, Computers for the local library, and Fixing potholes on city streets? $\endgroup$ – BruceET Sep 2 '18 at 19:25
  • $\begingroup$ @BruceET Hmmm....more evidence surveys are like a form of art in a way I suppose. There are no rules. $\endgroup$ – Taal Sep 15 '18 at 4:47
  • 1
    $\begingroup$ There are (almost) no rules if you just want to give a survey. There some rules if you want it to yield useful information: Subject must understand question, Subject must know answer to question, Subject must be willing to share that information truthfully. Subject must be able to provide information in a way that is useful to investigator. It's a chain; any missing link and investigator loses. $\endgroup$ – BruceET Sep 15 '18 at 7:41
  • $\begingroup$ @BruceET Agreed. There are rules. There are also rules to be able to paint a picture. Subject needs paintbrush. Subject needs paint. Subject needs canvas. Subject does not need to know how to paint though...or do they? Subject doesn't need to know the answer to the question actually either (likert is subjective) a lack of information is information itself. Subject's honesty is, in a way, part of the survey as (or experiment) as well too when compared against other subjects. I believe it's one of those places where science, math, and "art" overlap. Fun to write them tho, heh. $\endgroup$ – Taal Oct 17 '18 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.