3
$\begingroup$

I'm working on an application similar to this one, where the intent is to sort a list of items with ratings according to the best estimate of their average rating. The solution proposed in this link, which may be familiar to many, looks like this:

$(WR) = \frac{vR + mC}{v+m}$

where:

  • $R$ = average rating (mean)
  • $v$ = number of ratings
  • $m$ = parameter signifying the strength of the prior
  • $C$ = mean rating over all items

While I can subjectively choose my preferred value of $m$, it seems to me fairly arbitrary. Is there some principled approach for determining this value?

$\endgroup$
3
$\begingroup$

Yes there is! And it happens to be the name of this site, cross validation!

Why are we using these values to rank the items as opposed to the original average ratings? Because we believe that they will better predict future ratings. So let's try to choose $m$ in such a way that we achieve the best predictions.

We can hold out a test set, compute these modified average ratings, and test how well they predict future ratings by means of some evaluation function. But what if the test set we hold out is not particularly representative of the population? We can repeat this procedure over several folds so that every rating appears in this test set exactly once.

If you don't have access to the original data, only average values and total ratings, then you might be out of luck, but could try simulating data, but if we knew how much ratings varied from the true average rating we wouldn't be having this discussion.

edit: If we knew how much ratings varied from the mean we would still be having this conversation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.