I have a 70 or so items all measured with a 5 point likert scale. What I'd like to do is take their mean value and rank the items according to their means but with respect to statistical significance in terms of differences in means. So items that do not have statistically significant means get the same rank.

Item - Likert Mean - Rank
A - 5 - 1
B - 4 - 2
C - 3.1 - 3
D - 3.099 - 3
E - 2 - 4
F - 1 - 5

How would I go about doing this?

  • 1
    $\begingroup$ Of what are you taking a mean? The 70 likert items? How large is your sample of means? $\endgroup$ – Placidia Jun 3 '13 at 1:28
  • $\begingroup$ sample is about 200. the mean is the mean value of the item measured with a likert scale. $\endgroup$ – incognito2 Jun 3 '13 at 1:45
  • 1
    $\begingroup$ What do you do about the potential problem of not being able to distinguish say C from D nor D from E, but C is significantly different from E. What do you do then? $\endgroup$ – Glen_b -Reinstate Monica Jun 3 '13 at 3:05

I have a bad feeling about this; but feelings notwithstanding, let's try a model.

Since you are averaging the ranks of 70 items, I think we can safely view your response as a continuous variable, and probably normal. If the population mean for this situation is $\mu$, then each observation has model $x_i=\mu + \epsilon_i$ and the only difference between two observations $x_1$ and $x_2$ is independent random error, with variance $2 \sigma^2$. With respect to the model, there is no meaningful difference between $x_1$ and $x_2$. Statistically, we would regard the difference between them to be significant only if $x_1-x_2$ is inconsistent with a normal distribution having mean 0 and variance $2 \sigma^2$.

But you don't have 2 observations; you have 200. If you rank your 200 observations, what would you expect to find? This question was posed by Kolomogorov and Smirnov and the answer is the k-s test statistic. The k-s basically looks at the biggest difference between the sample and theoretical CDF's. In a similar way, you could look at jumps between your order statistics (or the sample CDF) and a normal distribution to find regime change points.

A simpler approach would be to treat this like a one-dimensional k-means clustering problem -- find your clusters and assign 1 to members of the smallest cluster; 2 to the members of the next cluster; and so on.

However this is not the kind of solution you wanted. Your probably wanted an answer where the number of distinct ranks is almost equal to the number of items. In other words, you are assuming a model of $x_i=\mu_i+\epsilon_i$, where occasionally, $\mu_i=\mu_j$. But that model has way too many parameters. You can't distinguish the $\mu$'s from the $\epsilon$'s. Any sensible measure of dispersion will ensure that most subjects are statistically indistinguishable from the others -- and the number of distinct ranks (i.e. clusters) will be small.

I can't see any good way of doing this. And I agree with @Glen_b's comment -- if your scheme isn't transitive, it won't work. But if it is transitive, you're basically doing clustering.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.