5
$\begingroup$

I need to work out a "average" (for lack of knowing a better word) of ratings or perhaps I could call them labels. Basically, I have a list of words that have been rated 1 - 3 for difficulty. 1 being easy, 2 moderate and 3 difficult. The list is marked by 5 individuals. I need to use the words that are "mainly" rated as 2. So the individuals did not always agree on the difficulty. If a word had the following "score": 1, 1, 2, 3, 3 then it's safe to say that it's agreeably a 2.

example of data

If most reviewers rated a word as 3 then it's obviously not on average a 2. Basically, when taking 5 reviewers' rating for a word, what is the average rating?

Now it makes sense that I want to add it all up and divide it by 5. This will mostly give me decimal values such as 1.6 or so on.

Now my question finally is. If doing so, getting values such as 1.6, what do I make of it? Round it to the nearest integer and take that as the deciding rating? Is it as simple as that?

$\endgroup$
6
6
$\begingroup$

This is largely an issue for you to decide based on your theoretical assumptions about the data and what lies behind them. When you calculate an arithmetic average, you are assuming that the intervals are reasonably similar. (That is, you are implicitly stating that $3-2 = 2-1$ and $3-1 = 2\times (3-2)$.) If you believe that is a reasonable assumption, and others in your field (e.g., reviewers) are likely to agree with you, then it's fine. Using means with ordinal data tends to be more defensible when:

  1. There are a larger number of ordinal levels (a rule of thumb is $\ge 12$);
  2. the ordinal levels are composed of many components (e.g., ratings for many related questions are aggregated into a composite); and/or
  3. the raters were instructed / tried to make the ratings equal interval.

It isn't clear to me that those hold in your case, but it is for you to decide.

You also should think hard about what you mean by '"mainly" rated as 2'. Again, that is for you to decide. However, I would not think of the set of ratings $\{1,1,2,3,3\}$ as "mainly" being $2$, despite the fact that the mean is $2$. I would interpret that as being a somewhat polarizing word, with some thinking it's 'easy' and some thinking it's 'hard'. But again, this is a theoretical issue for you to decide.


For what it's worth (almost certainly very little), if it were me, I would think your ratings were not amenable to be described by means. I think I would interpret '"mainly" rated as 2' as the majority of raters gave this word a 2. That is, I would select words that received $>50\%\ \rm ``2\!"$s.

By contrast, I suspect that you don't only want to select individual words '"mainly" rated as 2', but also want the entire set of selected words to be rated $\approx 2$. To check that aspect, I would feel more comfortable using the mean of all the ratings for all the selected words (or the mean of the words' means). At this point, you are averaging over many more ratings and I think the mean would be more defensible.

$\endgroup$
3
  • $\begingroup$ I am in complete agreement with your latter two paragraphs. One needs to be careful about having a bunch of 1's and 3's that balance out to a two. $\endgroup$ – Kitter Catter May 25 '16 at 19:49
  • $\begingroup$ @KitterCatter, yes, I noticed after posting that you had made the same point already at the end of your answer (+1). $\endgroup$ – gung - Reinstate Monica May 25 '16 at 20:09
  • $\begingroup$ Thank you, both! @KitterCatter and @gung! I am opting for the > 50%. I think your advice about 1's and 3's balanced as a 2 not being quite right is good advice. I must say, I learnt quite a lot through all the answers and comments! $\endgroup$ – amateurjustin May 26 '16 at 14:10
3
$\begingroup$

It is often the case with discrete data that the mean is non-discrete. That doesn't mean that it isn't a worthwhile statistic to report. A value of 1.6 could be interpreted as easy for most, but intermediate for some - pretty much what you see in your table.

If you want integers, you can calculate the median, which is the value such that half of all observations are below it and half above it. For example for kanarie, the median of (1,1,2,2,2) is 2, since half are below 2 and half are above (or equal).

Alternatively is the mode, which is just the most common value. It is often useful to get a feel for the overall population's choice.

$\endgroup$
8
  • 1
    $\begingroup$ (+1) It's worth considering, though, whether you want to treat easy-moderate-difficult as a interval scale in the first place. Taking means implies you're happy to say that the difference between easy & moderate is equal to the difference between moderate & difficult. $\endgroup$ – Scortchi - Reinstate Monica May 25 '16 at 17:31
  • 1
    $\begingroup$ @PeterK. you could round the median in the same way that the OP suggested rounding the mean -- my gut tells me that would be "less wrong" but I don't have a good justification for it $\endgroup$ – shadowtalker May 25 '16 at 17:37
  • 1
    $\begingroup$ [link]stats.stackexchange.com/questions/31598/… seams to get to the same problem $\endgroup$ – amateurjustin May 25 '16 at 17:54
  • 2
    $\begingroup$ No; with 1,1,2,2,3 there are two modes, 1 and 2. This is either a limitation of the mode that it's often not unique or just a feature of data like this. Or both. Positively put, the mode is most useful when it's well defined. See also @Martin's answer. $\endgroup$ – Nick Cox May 25 '16 at 18:24
  • 1
    $\begingroup$ I'd suggest that there are other answers that can be considered reasonable depending on what exactly you are trying to do. Using median may lead to some polarized results being represented as in the middle which could be bad for your use. $\endgroup$ – Kitter Catter May 25 '16 at 20:30
2
$\begingroup$

You state that you "need to use the words that are 'mainly' marked as 2."

I'd argue that you need to define what 'mainly' actually means to you.

One way is to take the mean, as you indicate intuitively makes sense to you. If you do, then, indeed, if the mean is: 1.5 ≤ mean < 2.5. Then you might interpret it as 'mainly' 2.

An alternative is to use the mode, as @ssdecontrol suggested. You would, however, need to decide on how to handle a situation where there are two modes: e.g. (1,1,2,2,3) where the modes are 1 and 2. Would you consider this to pass your criteria? Or your example of (1,1,2,3,3) where the modes are 1 and 3. The mode is not 2 in this case, but the mean is exactly 2. Does it pass your criteria?

A third option is to use the mean of the mode. Where the pass criterion could be: 1.5 ≤ mean of mode < 2.5.

$\endgroup$
2
  • $\begingroup$ I think that might be the best option yet! The mean of the mode! $\endgroup$ – amateurjustin May 25 '16 at 18:09
  • $\begingroup$ Is "the mean of the mode[s]" commonly used? $\endgroup$ – underscore_d May 25 '16 at 22:49
1
$\begingroup$

I think you have a few options depending on what you care about. You can go with the mean and see if it falls within some range of 2. This would be like averaging your votes. The advantage here is that you can check on how balanced people's votes are.

If you care more about how the votes land with at least 1/2 voting more or less at some point you would want to go with the median. The advantage of the median is that it is based more on vote distribution.

You might be interested in where the most voters voted. This would be the mode. You used the word mainly so you might be interested in this metric. Advantage is where the most voted, disadvantage is that as you get more options this can mean less.

One thing to keep in mind is also that you may have some controversial words where most voters are 1 and 3. In this case you should be careful since most would not say 2, but some methods would give you 2. I would thus suggest you use mode, in the case of two modes you would reject or use another method such as median.

e: You could also look into percent of votes that are 2. If it is above 50% then mainly people believe the question is a 2.

E: If you really want to add complexity you could also try to address the issue of voters not agreeing and having a bias. You would then need to do some transformation of the vote and do a mean/median.

$\endgroup$
2
  • $\begingroup$ "[mean] would be like averaging your votes." Sorry for the basic interjection, but in all my classes mentioning stats, pretty much the first thing they drummed into us was that the mean is just one type under the umbrella category of averaging methods, and that using the latter when we meant the former was a rookie mistake from colloquial use. Is this a regional difference? $\endgroup$ – underscore_d May 25 '16 at 22:53
  • $\begingroup$ I wrote this for a lay audience so I used the colloquial terminology. $\endgroup$ – Kitter Catter May 26 '16 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.