Using Standard Deviation to Understand Consensus I was recently asked a question, and while I think I know the answer intuitively, I can't explain it.

Scenario:
  A questionnaire of 20 questions was given to 6 people. Each person was asked to answer each question by ranking it on a scale of four numbers: 1, 2, 3, 4, 5.
The questionnaire was then analyzed, and, for each question, the standard deviation was calculated using the 6 people's rankings of 1-4. This standard deviation was then called the "consensus" of a question. All the consensus numbers were then ranked against each other and those with high standard deviations were considered to be "high consensus."

My Issue:
This doesn't make any sense to me.... at all. The standard deviation is a measure of "how much the data varies." If you look at a single point and compare it to the standard deviation, it should tell you how close to "normal" it is (for the data set you are analyzing).
A better measure of consensus would be to just add up how many people voted similarly for a single question (i.e. if 4 of the 6 people voted the same rank for one question, then there is high consensus).
Am I crazy, or does the standard deviation method actually work and I'm just not understand it?????
 A: I suggest that the standard deviation and consensus are INVERSELY proportional. Because standard deviation measures the amount of difference in the answers, it seems logical that the greater the difference (i.e. the higher the standard deviation) then the lower the consensus.
Does that make sense?
A: The description of this questionnaire is a bit unclear. If respondents were asked to make an ordinal rating of each item, and one was trying to compare the level of consensus for each item, then one could use a weighted version of Cohen's Kappa. That statistic is used to demonstrate the level of agreement between different raters on binary items. Weighted versions exist for ordinal items. Basically, you would weight agreement as 0, and disagreements as something other than 0 - usually 1, but you might want to increase the weight to more than one if you want to penalize larger disagreements more. I am not familiar with weighted Kappa, and potential users would want to do some homework on an appropriate weighting scheme.
When reporting the results, I would urge users to say what scheme they used, and why they chose it. I'm not clear that there is a universally accepted default weighting.
