I'm sure there's a more technical term for what I'm looking for, but I don't know what it is (it's not confidence intervals, I think)

I'm trying to make a system where people get ratings (from 1 to 5 stars) over time, and I want to combine all those ratings into one individual score for each person. Now, this score alone is not a very good thing, I'd like to have a second value that shows how "reliable" / "accurate" this score is.

Let's say that:

  • If someone gets 1 rating of "3", his score is "3", but we're not very sure of that.
  • If someone gets 10 ratings of "3", we're very confident he's a "3". Let's say that more than 10 ratings that are identical are not a stronger signal than 10 ratings, there's a point where we're "confident enough". Let's call this 100% confident.
  • If someone gets 2 x"1", and 2 x "5", the average is also a 3, but the person is all over the place, and I want to reflect this by giving this score a very low confidence value.

So, I'm pretty much looking for something that reflects variance, but also reflects how many "consistent" ratings someone has got. Or how "stable" he is at his score.

Can anyone think of either an algorithm for doing something like this, or point me in the direction of tools that you'd use for such a thing?

Thank you!


Evan Miller's how not to sort by average rating has pretty good advice for the general problem you are trying to solve.

  • $\begingroup$ I was going to answer the same thing. I would fit a distribution and use the lower bound of the confidence interval for the ranking. Unfortunately, for example a t-distribution parametrized on the sample mean and variance. However, I think you need a minimum number of votes for each item. $\endgroup$ – Simone Mar 7 '13 at 23:04
  • $\begingroup$ You can have a look here stats.stackexchange.com/questions/37993/… about the minimum sample size. I would really be interested to see if there is any other distribution that we can fit with only two samples or at the limit just one. $\endgroup$ – Simone Mar 7 '13 at 23:11

What you are looking for is in fact exactly a measure of the variance of the ratings for each person - as you get very close to in your own wording of the question. You might want to call this the intra-subject variance. You could use either the original variance, or its square root, which turns it into the standard deviation and has the advantage of being directly comparable to the original scale of 1-5.

Because you can conceptualise your ratings as samples from a hypothetical infinite population of ratings for each subject, and it sounds like different subjects have different numbers of ratings, it's probably quite important to use the sample variance rather than the population variance. This gives you an unbiased estimate of the population variance, taking into account that before estimating the variance you have first had to estimate the mean.

  • $\begingroup$ Yes, except for the fact that a person with 1 "3", and a person with 100 "3"'s will both have a variance of 0, and the second one should score much higher. $\endgroup$ – Daniel Magliola Mar 11 '13 at 13:56
  • 1
    $\begingroup$ The person with only one 3 will have an undefined sample variance ie once a degree of freedom is spent on estimating the mean. But your point remains a good one if comparing a person with 2 "3"s and another with 100 "3"s. I think a good fix for that might be moving to a Bayesian paradigm. $\endgroup$ – Peter Ellis Mar 11 '13 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.