1
$\begingroup$

I'm sure there's a more technical term for what I'm looking for, but I don't know what it is (it's not confidence intervals, I think)

I'm trying to make a system where people get ratings (from 1 to 5 stars) over time, and I want to combine all those ratings into one individual score for each person. Now, this score alone is not a very good thing, I'd like to have a second value that shows how "reliable" / "accurate" this score is.

Let's say that:

  • If someone gets 1 rating of "3", his score is "3", but we're not very sure of that.
  • If someone gets 10 ratings of "3", we're very confident he's a "3". Let's say that more than 10 ratings that are identical are not a stronger signal than 10 ratings, there's a point where we're "confident enough". Let's call this 100% confident.
  • If someone gets 2 x"1", and 2 x "5", the average is also a 3, but the person is all over the place, and I want to reflect this by giving this score a very low confidence value.

So, I'm pretty much looking for something that reflects variance, but also reflects how many "consistent" ratings someone has got. Or how "stable" he is at his score.

Can anyone think of either an algorithm for doing something like this, or point me in the direction of tools that you'd use for such a thing?

Thank you!
Daniel

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Evan Miller's how not to sort by average rating has pretty good advice for the general problem you are trying to solve.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I was going to answer the same thing. I would fit a distribution and use the lower bound of the confidence interval for the ranking. Unfortunately, for example a t-distribution parametrized on the sample mean and variance. However, I think you need a minimum number of votes for each item. $\endgroup$
    – Simone
    Commented Mar 7, 2013 at 23:04
  • $\begingroup$ You can have a look here stats.stackexchange.com/questions/37993/… about the minimum sample size. I would really be interested to see if there is any other distribution that we can fit with only two samples or at the limit just one. $\endgroup$
    – Simone
    Commented Mar 7, 2013 at 23:11
0
$\begingroup$

What you are looking for is in fact exactly a measure of the variance of the ratings for each person - as you get very close to in your own wording of the question. You might want to call this the intra-subject variance. You could use either the original variance, or its square root, which turns it into the standard deviation and has the advantage of being directly comparable to the original scale of 1-5.

Because you can conceptualise your ratings as samples from a hypothetical infinite population of ratings for each subject, and it sounds like different subjects have different numbers of ratings, it's probably quite important to use the sample variance rather than the population variance. This gives you an unbiased estimate of the population variance, taking into account that before estimating the variance you have first had to estimate the mean.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, except for the fact that a person with 1 "3", and a person with 100 "3"'s will both have a variance of 0, and the second one should score much higher. $\endgroup$
    – Daniel Magliola
    Commented Mar 11, 2013 at 13:56
  • 1
    $\begingroup$ The person with only one 3 will have an undefined sample variance ie once a degree of freedom is spent on estimating the mean. But your point remains a good one if comparing a person with 2 "3"s and another with 100 "3"s. I think a good fix for that might be moving to a Bayesian paradigm. $\endgroup$
    – Peter Ellis
    Commented Mar 11, 2013 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.