2
$\begingroup$

Comparing ratings for recipes I have often wondered is, say, 50 reviews of average 4.5 better than 31 reviews of average 4.8?

Orbiting these considerations has been some idea of statistical significance.

All uncertainty threatens to be dispelled today since I decided to ask here. No doubt there is a clear answer to the question, is it possible to determine simply from the number of ratings (and their average, or even their distribution), which recipe is better? And if so, what is a quick formula to mentally evaluate this?

Edit: To provide some more clarity, a definition of 'better':

  1. All reviews are rated on the real interval $[0,5]$. A recipe is modelled as having some real, independent, "objective" "score" (on the same interval) that is estimated, with error, by the ratings of the recipe reviewers. It is to this "essential" score we refer when we assert one recipe is better in comparison to another. We attempt to access this essential score through models of it, such as through the average ratings, and number of ratings of that recipe.
  2. If the ratings (of two recipes $F$ and $G$ for the same enreciped menu item, $M$) are on the same site, and have the same number of reviews, then barring the inclusion of any additional information, recipe $F$ is better than $G$ if $avg(ratings(F)) > avg(ratings(G))$.
  3. If the ratings are on the same site, and have different numbers of reviews, I don't know.
  4. If the ratings are on different sites, then, barring the inclusion of any additional information, as per 2 and 3.
  5. If additional information is available, such as, for example (and readers may choose to delve as deeply as they like in this direction):
    • the reliability of site $A$ versus site $B$. This is left to the reader's discretion, and it is imagined a probability value giving the "confidence of belief in the reliability of" site $A$'s ratings, that is, a measure of site $A$'s rating's fidelity to the underlying "true" value of the recipe's rating. So, if site $A$ provides very high ratings to recipe of an item $M$, but is generally thought to be unreliable signal of betterness, while site $B$ provides somewhat lower average, with much greater reliability, to a different recipe of the same item $M$, one could make a reasonable case that site B's recipe is better than site $A$'s. Any metric of site reliability and associated mechanism of computing an ordering on recipe's must reach the same conclusion.
    • the distribution of reliability of the raters, that is, a measure of how closely a rater's score for any recipe, say $score(rater, recipe)$, models the recipe's underlying "objective" score.

In terms of what's required to answer this question, 3 is necessary and sufficient, and 5 is unnecessary and (alone) insufficient, although welcome.

$\endgroup$
3
  • 2
    $\begingroup$ The clear answer will be somewhere along the lines of 'better is what you define it to be', I'm afraid. $\endgroup$ Commented Jul 25, 2014 at 14:06
  • $\begingroup$ one of many possible answers is discussed in detail at nbviewer.ipython.org/github/CamDavidsonPilon/… $\endgroup$
    – Jacob
    Commented Jul 25, 2014 at 14:54
  • $\begingroup$ So again is there a clever little equation in $numrating$ and $avgscore$ that permits you to compare and order the pairs ($numrating_i$, $avgscore_i$) by which has the higher "objective" rating of a recipe (which these pairs model), in a way that an argument could be made for such an equation's close modelling of an average human's intuition about the order of such pairs? I take it Marc does not know one. In case there is no such simple answer, please see the above clarity I have imbibed into the work. $\endgroup$
    – Cris
    Commented Jul 25, 2014 at 23:12

1 Answer 1

2
$\begingroup$

As Marc says, you first need to define "better". "Statistical significance" is only applicable if you want to say whether the 4.5 and the 4.8 averages are statistically significantly different from each other. (It may be that the amount of information you have, and the kind of information, doesn't allow you to actually say that the two recipes do in fact have different average ratings.)

It gets even more complicated: "statistically significantly different" is not "practically significantly different".

Are the ratings on the same website or on two different sites? If they're on two different sites, the raters and therefore how hard or easy they rate things could be completely different. Even on the same website, different recipes might appeal to very different cooks who rate on different scales. What about time: what if one recipe was mostly rated 5 years ago when it first came out, but the other is mostly rated in the last six months? Does that matter to you? What if one recipe is considerably harder to get right? (So that average cooks are more likely to mess it up or figure it's too hard.) Etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.