2
$\begingroup$

I was recently asked a question, and while I think I know the answer intuitively, I can't explain it.

Scenario: A questionnaire of 20 questions was given to 6 people. Each person was asked to answer each question by ranking it on a scale of four numbers: 1, 2, 3, 4, 5.

The questionnaire was then analyzed, and, for each question, the standard deviation was calculated using the 6 people's rankings of 1-4. This standard deviation was then called the "consensus" of a question. All the consensus numbers were then ranked against each other and those with high standard deviations were considered to be "high consensus."

My Issue: This doesn't make any sense to me.... at all. The standard deviation is a measure of "how much the data varies." If you look at a single point and compare it to the standard deviation, it should tell you how close to "normal" it is (for the data set you are analyzing).

A better measure of consensus would be to just add up how many people voted similarly for a single question (i.e. if 4 of the 6 people voted the same rank for one question, then there is high consensus).

Am I crazy, or does the standard deviation method actually work and I'm just not understand it?????

$\endgroup$
  • $\begingroup$ I think you want to compare variances across questions. For example, say 2 people filled out the survey and q1 is (1,5) and q2 is (3,3). The variance of the first question is 8 and the second is 0, so we might say there is less consensus on the first. $\endgroup$ – Dimitriy V. Masterov Jun 8 '15 at 23:24
  • $\begingroup$ @DimitriyV.Masterov But consider for five people: q1 is (1,1,2,3,4) with sd of 1.3; q2 is (1,1,1,1,2) with sd of 0.4; q3 is (1,1,4,4,4) with sd of 1.6; q4 is (3,3,3,4,4) with sd of 0.5; This implies the consensus is q2 < q4 < q1 < q3, but that's not true. It should be q1 < q4 <= q3 < q2. The standard deviation is combining the number of votes with the scale of the vote, but that doesn't show consensus. $\endgroup$ – Very Confused Jun 8 '15 at 23:50
  • $\begingroup$ Please add the [self-study] tag & read its wiki. $\endgroup$ – gung - Reinstate Monica Jun 9 '15 at 0:01
  • $\begingroup$ @gung This is not a question from a book or for a course. This is a real life scenario I'm currently faced with. A coworker created a mess of a document, and I'm trying to sort through it. I feel like my logic is correct and that the standard deviation is being incorrectly used. Given that, I'm happy to add [self-study] if you feel it is appropriate, but I'm really trying to just work through this logic. $\endgroup$ – Very Confused Jun 9 '15 at 0:08
  • $\begingroup$ @VeryConfused I would not defend this method very hard, but that seems to be what you colleague is doing (other than the part about high variance being high consensus and why only 1-4). In any case, your ranking is not entirely obvious to me either. $\endgroup$ – Dimitriy V. Masterov Jun 9 '15 at 0:52
1
$\begingroup$

I suggest that the standard deviation and consensus are INVERSELY proportional. Because standard deviation measures the amount of difference in the answers, it seems logical that the greater the difference (i.e. the higher the standard deviation) then the lower the consensus.

Does that make sense?

$\endgroup$
  • $\begingroup$ I agree, whoever wrote that paragraph probably made a typo (or a think-o) and the last sentence should read "...those with low standard deviations have high consensus" $\endgroup$ – bdeonovic Jul 1 '19 at 21:52
0
$\begingroup$

The description of this questionnaire is a bit unclear. If respondents were asked to make an ordinal rating of each item, and one was trying to compare the level of consensus for each item, then one could use a weighted version of Cohen's Kappa. That statistic is used to demonstrate the level of agreement between different raters on binary items. Weighted versions exist for ordinal items. Basically, you would weight agreement as 0, and disagreements as something other than 0 - usually 1, but you might want to increase the weight to more than one if you want to penalize larger disagreements more. I am not familiar with weighted Kappa, and potential users would want to do some homework on an appropriate weighting scheme.

When reporting the results, I would urge users to say what scheme they used, and why they chose it. I'm not clear that there is a universally accepted default weighting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.