6
$\begingroup$

I'm a complete newbie to statistics (although I find it really interesting!), and I have taken the task of distributing feedback to speakers of a conference I'm co-organizing. Each speaker was given a grade on the scale 1-5 from the participants, and we combine feedback from all participants into a mean score, for instance 3.56. We can then order the speakers by mean score.

In addition to giving the speakers their mean score, we also want to give them a clue about how they did compared to the other speakers. To avoid discouraging the speakers who did the worst (we want people to try again!), we came up with giving back a more fuzzy metric. We want to divide the speakers into four groups: 10% best/20%/30%/40% worst. Does there exist a name for this?

EDIT: I'll try an example. If I have ten talks with scores of { 1.2, 1.3, 2.1, 2.4, 2.7, 3.0, 3.2, 4.1, 4.2, 4.5}, I would divide them up like this:

10% best: 4.5
20% "next best": 4.2, 4.1
30% "next worst": 3.2, 3.0, 2.7
40% worst: 2.4, 2.1, 1.3, 1.2

What I want to know is if there is a name for these ranges.

Improve this question
$\endgroup$
2
  • $\begingroup$ Please pay close attention to dmk38's comments, below. Who is rating each speaker, and how much variability there is in each speakers' ratings make a difference. That 4.5-rated person has a higher mean than someone rated, say 4.3, but depending on whether their audiences had significantly different personal grading criteria and how much variation in grades there was around each mean, there may be no real-life difference between the two. The numbers will look authoritative, since you ran them through a computer, but the question is: are they saying what you think they're saying? $\endgroup$
    – Wayne
    Dec 21, 2010 at 17:32
  • $\begingroup$ Thank you for all the help, people. I really appreciate it! Also, thanks for pointing out the drawbacks with this way of measurement. I have lots to learn about this stuff. $\endgroup$
    – mranders
    Dec 22, 2010 at 20:27

4 Answers 4

Reset to default
9
$\begingroup$

If I understand you correctly, you may refer to Percentiles, perhaps espacially Quartiles.

Perhaps you can elaborate a little more on which percentages should be enclosed in each bin, to get a more accurate answer.

UPDATE: Based on the comments below decile seems to be the term you want. For your data these can easily achieved via R (note the differences from your example):

> x = c(1.2, 1.3, 2.1, 2.4, 2.7, 3.0, 3.2, 4.1, 4.2, 4.5)
> quantile(x,c(0.9,0.8,0.7,0.6))
 90%  80%  70%  60% 
4.23 4.12 3.47 3.08
> x[x > quantile(x,0.9)]
[1] 4.5
> x[x > quantile(x,0.8) & x < quantile(x,0.9)]
[1] 4.2
> x[x > quantile(x,0.7) & x < quantile(x,0.8)]
[1] 4.1
> x[x < quantile(x,0.7)]
[1] 1.2 1.3 2.1 2.4 2.7 3.0 3.2

You can then tell the speakers you are in the 10th decile, 9th decile, 8th decile, or 7th and worse decile (or something with above the 90th percentile, ...). But from my point, the problem will always be to name the catch all (i.e. worst) category.

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ In particular, 10%, 20%, 30% etc. are called Deciles. $\endgroup$
    – nico
    Dec 21, 2010 at 13:41
  • 1
    $\begingroup$ Deciles are then what you need. Households for example are divided into deciles according to their income. Poorest are in the first, richest in tenth. $\endgroup$
    – mpiktas
    Dec 21, 2010 at 14:20
  • 3
    $\begingroup$ @mpiktas Right. So to give a clear, final answer to the original question, we might say the first category is the "top (i.e., tenth) decile," the next category consists of the "eighth and ninth deciles," the third category is the "fifth through seventh deciles," and the last category contains the "lowest four deciles" (or, to be more circumspect, is "not in the top six deciles" or even "did not make it in the top 60%"). $\endgroup$
    – whuber ♦
    Dec 21, 2010 at 14:52
  • 1
    $\begingroup$ But the wikipedia article states that a decile is "any of the nine values that divide the sorted data into ten equal parts, so that each part represents 1/10 of the sample or population" - thus a single value, not a range. The R function ´quantile´ also returns a single value. Is it not more correct to say for instance "between the 8th and 9th decile"? $\endgroup$
    – mranders
    Dec 21, 2010 at 19:29
  • 1
    $\begingroup$ The decile value is just the border of the range. Values higher are within this range (until the next decile). $\endgroup$
    – Roman Luštrik
    Dec 22, 2010 at 9:52
3
$\begingroup$

You have the answer you asked for, but along with how to communicate this information you might also want to think about how to asses the reliability & precision of the scores. If the evaluators aren't really using the same standards, the scores will furnish a misleading measure of the quality of the speakers no matter how you decide to categorize them. Also, even if the evaluators are reliable in this sense, your rankings should be sensitive to the standard error in their measurements (likely to be large if you have only a modest number of evaluators): you don't want to imply that there are meaningful differences among speakers whose scores differ by amounts that are comparable to the level of background noise in your data. If you aren't in a position to furnish genuinely informative quantitative feedback, you are better off, in my view, picking one or two evaluators you trust to give the speakers' qualitative feedback informed by the evaluators' own observations & by their assessments of whatever evidence they have on the reactions of others.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The conference had almost 500 participants, at all time divided into three tracks. Not all of our participants gave us ratings, of course, not all participants were at a talk at all times, and they were not evenly distributed among the tracks. I'd say we have about 50% feedback rate, based on the numbers. Some tracks has as much as ~89 feedbacks, some as low as ~22. Since I do not have full access to the raw data, I cannot do stuff like eliminate outliers etc. When will the data be unsafe to draw conclutions from? $\endgroup$
    – mranders
    Dec 21, 2010 at 19:45
  • $\begingroup$ I don't grasp the layout fully. But if you had groups of evaluators judging multiple speakers, you should perform an appropriate inter-rater reliability test (e.g., ICC) for each group. If that test shows sufficiently high agreement among evaluators--and you believe they were competent to judge (presumably; why else ask? still you are assuming the validity of your measures)--compute the mean scores & standard errors for each speaker. You must use judgment at that point to decide how much imprecision you are comfortable with in making your rankings--much as you would when grading on a curve. $\endgroup$
    – dmk38
    Dec 21, 2010 at 20:42
3
$\begingroup$

In addition to the existing answers, you may find it useful to read up about test norms, a well established topic in psychology and education.

Improve this answer
$\endgroup$
0
1
$\begingroup$

I'd like to add another cautionary note, and a suggestion. When asked for a 1-5 rating, I usually think up some scale, like:

  1. worst ever
  2. real bad
  3. ok
  4. great
  5. awesome

If your raters did similarly, taking an average is somewhat questionable; the difference between "worst ever" and "real bad" may be larger than the difference between "great" and "awesome". In jargon, your data may be somewhere between ordinal and interval (see http://en.wikipedia.org/wiki/Level_of_measurement).

For this reason, I'd suggest that you give your speakers a histogram of their ratings as part of their feedback. And maybe even rank speakers by percent of their ratings above, say, 3, rather than by their mean rating.

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ Not to mention that every individual rater probably creates their own scale, with their own variations on the differences between levels... this stuff is absolutely loathsome. Gary King's papers on anchoring vignettes in survey research provide a good overview of the problems here, and an interesting solution (that can't be applied to this particular case). $\endgroup$
    – Matt Parker
    Dec 22, 2010 at 20:08
  • $\begingroup$ Holy crap, this just gets worse... Unfortunately, due to the way we counted up the scores from the feedback forms, I do not have the data for how many of each score the speakers received. I think I will include a disclaimer in the feedback email, detailing some of the uncertainties of the data interpretation. $\endgroup$
    – mranders
    Dec 22, 2010 at 20:21
  • $\begingroup$ That's too bad. If it's any consolation, Amazon uses the average of 5-star ratings to rank their data (as one option), so you've got good company. $\endgroup$
    – XZVASFD
    Dec 24, 2010 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.