1
$\begingroup$

My respondents has to evaluate number of items (say, 20). Formerly they had to just check three that they liked (like/dislike) and I simply counted the most checked.

I 'improved' survey by replacing check-marks on rank (1-2-3) so respondents not just put check-marks but rank (1,2,3) three attributes they like. That is, every respondent has to choose three best items and rank them. Now I'm struggling how to account this.

Known "IMDB" formula based on Bayesian average seems doesn't work for me because puts too much weight on "unscored' items (unranked items considered 'not so bad' by default). In my situation, the fact that item was marked is much more important because respondents always evaluate ALL items (in contrast with IMDB films or any other rating where rating occurs by chance).

Improve this question
$\endgroup$
7
  • $\begingroup$ Your title seems a bit to contradict the text. You say they select-then-rank-it. But "stars" are usually used in rating (such as Likert), not ranking. $\endgroup$
    – ttnphns
    Commented Feb 24, 2015 at 13:24
  • $\begingroup$ Yes, they select three items then rate them 1*, 2* and 3*. Our goal is rank items based on the obtained ratings. $\endgroup$
    – Niksr
    Commented Feb 24, 2015 at 13:39
  • $\begingroup$ You say not just put check-marks but rank (1,2,3). Or rate? I selected 3 attributes, A, C, F. Can then there rating be 3, 3, 2. Or they must be 1, 2, 3 in some order? $\endgroup$
    – ttnphns
    Commented Feb 24, 2015 at 13:44
  • $\begingroup$ Within one evolution occurrence by one respondent the rank and the rate is the same. One chooses three items then rank them. Or rate on 1,2,3 scale. YES, they must be 1, 2, 3 in some order? $\endgroup$
    – Niksr
    Commented Feb 24, 2015 at 13:54
  • $\begingroup$ Not very instructive so far. Please, tell me: a respondent selected 3 attributes, does then he rate or rank the three? Ranking implies that he has to distribute 3 values (1, 2 and 3) - and only three - among the three attributes; any two attributes of the three cannot receive the same value. Is that the case or not? Please, edit your question to make the task very clear. $\endgroup$
    – ttnphns
    Commented Feb 24, 2015 at 14:02

1 Answer 1

Reset to default
3
$\begingroup$

There can be many ways of computing "rating" of attributes based on complete or incomplete ranking task. Each way implies its assumption or model, so it is difficult to say which one might be the "best".

Most natural seems to convert ranks into scores and do it non-linearly. Imagine that all 1 through 20 ranks are possible. We assume rank 1 is the "best" and rank 20 is the "worst". Intuitively, the greater in the rank number the lesser is its qualitative difference with the adjacent ranks. Like in sport: "distance" between gold and silver medals is the greatest; between silver and bronze is somewhat less; between bronze and 4th place is further diminished; between 10th and 11th places is hardly distinguishable. Thinking this way, any decreasing "exponentially fading" function of rank will basically do.

For example, I could recommend considering reciprocal function 1/Rank or Exponential function in the form of (backward) Savage scores. Both functions are shown below.

Rank  1/Rank    Savage (backward)
1     1.0000    2.5977
2      .5000    1.5977
3      .3333    1.0977
4      .2500    .7644
5      .2000    .5144
6      .1667    .3144
7      .1429    .1477
8      .1250    .0049
9      .1111    -.1201
10     .1000    -.2312
11     .0909    -.3312
12     .0833    -.4221
13     .0769    -.5055
14     .0714    -.5824
15     .0667    -.6538
16     .0625    -.7205
17     .0588    -.7830
18     .0556    -.8418
19     .0526    -.8974
20     .0500    -.9500

Savage (it's a name) scores are particularly interesting because of their properties. They have always mean 0 and variance greater as the number or data rows N (number of ranks 20, in this instance) grows. Whatever is N, the curve of fading is approximately the same. And the fading (the flattening) is slower than with 1/Rank function.

Savage score is computed as $S_i = (1/N + 1/(N-1) +…+ 1/(N-R_i+1)) - 1 $ with $R_i$ being the rank. In our example ranking is backward (that is, the largest $R_i=20$ corresponds to rank=1).

Since in the OP question the ranking was incomplete - only ranks 1, 2, 3 were used and ranks 4 through 20 were dismissed (say, the rating process was "interrupted") we have to assume some equal score for all the attributes not ranked by the respondent. The mean of scores for ranks 4 through 20 is the obvious candidate for the value. Another (and more risky) approach would be to impute the mean score corrected by expectation given the attributes that were selected (ranked) by the respondent. That is, if attributes A and B are often seen to be selected together then for those respondents who selected A but not B the score for B shall be higher than just the average score between scores 4 through 20.

In this example with the task "select three and rank them" , ranks 4 through 20 were dismissed. In the unconstrained task "select whatever applies and rank the selected" each respondent will have his own "tail" of dismissed ranks. You might then use the respondent-specific average to impute as the score for the not chosen items. However, in the unconstrained task it is natural to assume that the importance of the not selected items is zero or absolute minimal for the respondent. Then there should be used the minimal possible score, and not the average or such as above.

Improve this answer
$\endgroup$
4
  • $\begingroup$ May be just sum up products of number of chosen items and scores? For example, Item#3 was evaluated 34 times by 3*, 9 times by 2* and 0 times by 1*, so total score is 34*3+9*2+0*1=120. Sounds not super scientific but results appeal to me in terms as survey results perceived intuitively. $\endgroup$
    – Niksr
    Commented Feb 24, 2015 at 14:41
  • $\begingroup$ Basically, my idea is the same as you write. But I recommended not to adopt linear function because, for me, ranking task is not linear (it is of course an assumption). $\endgroup$
    – ttnphns
    Commented Feb 24, 2015 at 14:51
  • $\begingroup$ Looking on actual data I found that it seems that respondents did NOT "rank all 20 but record only top 3" as you assumed but in fact chose exactly 3. It means that every item has clear unimodal close to normal distribution of evaluations among 1-2-3 places (let's call it "places" to not confuse with rate and rank), so it pretty obvious (just looking on data) which items were candidates for 1st place, for 2nd and for 3rd place. I think we should rank every item on its own scale first (1st on 1st place scale, 2nd on 1st place scale, etc), then combine obtained ranks in some non-linear manner $\endgroup$
    – Niksr
    Commented Feb 25, 2015 at 5:34
  • $\begingroup$ @ttnphns please let me know if you know an answer for this: stats.stackexchange.com/questions/403715/… Looking forward to hearing from you :) $\endgroup$
    – EmJ
    Commented Apr 18, 2019 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.