0
$\begingroup$

I’m building a movie review aggregation site that combines user ratings and critical reviews for a given movie. The objective is to create a list of the “best of the best” movies which were rated highly by the general audience and which were hailed by critics as one of the best movies of the year.

I have two data sets:

  1. Audience Ratings: Scored from 1 - 5 stars. Due to the limitations of the data set I'm using, the standard deviation of audience ratings for a given film is unknown.

  2. Critical Ratings: Many critics don’t provide quantifiable ratings (i.e. letter grade, star rating) in their reviews, and I don’t want to read through hundreds of reviews and hand code them myself (which is what Metacritic does). As a proxy for critical reception, I’m looking at yearly “Best Movie” lists, and scoring movies based on how many “Best Movie” lists they appeared on in a given year. For example, if there are 20 “Best movies of 2021” lists and a movie appears on all 20 lists, then it would receive the highest score, whereas if a movie didn’t appear on any Best Movie lists at all it would receive the lowest score.

My question: is there any way to combine the user ratings and critical ratings into a single composite score in a way that isn't completely arbitrary? I see most review aggregator sites (e.g. Rotten Tomatoes, Metacritic) keep audience and critical scores separate, but I wanted to hear people’s ideas on whether there is a statistically valid way to create a composite score to create the most simple user experience possible.

What I've already tried

One approach is to normalize both the Audience Rating and the Critical Rating, and then creating a weighted average of the two, placing much more weight on the Audience Rating because it encompasses far more data points. This weighted average would serve as the Composite Score for a given movie and would be the basis of movie rankings for a given year, genre, etc.

Here’s an example using Mad Max: Fury Road and Moonlight:

Mad Max Fury Road Audience Rating: 4.6 / 5

Mad Max Fury Road Critical Rating: Appears on 7 / 15 “Best Movies of 2015” lists

Mad Max Normalized Audience Rating: (4.6 - 1) / (5 - 1) = 3.6 / 4 = 0.9

Mad Max Normalized Critical Rating: (7 - 0) / (15 - 0) = 7 / 15 = 0.47

Weighted Average (giving 90% weight to Audience Rating at 10% weight to Critical Rating): (0.9 * 0.9) + (0.47 * 0.1) = 0.81 + 0.047 = 0.857

Moonlight Audience Rating: 4.3 / 5

Moonlight Critical Rating: Appears on 12 / 15 “Best Movies of 2016” lists

Moonlight Normalized Audience Rating: (4.3 - 1) / (5 - 1) = 3.3 / 4 = 0.825

Moonlight Normalized Critical Rating: (12 - 0) / (15 - 0) = 12 / 15 = 0.8

Weighted Average (giving 90% weight to Audience Rating at 10% weight to Critical Rating): (0.825 * 0.9) + (0.8 * 0.1) = 0.7425 + 0.08 = 0.8225

$\endgroup$

1 Answer 1

0
$\begingroup$

The weighted average is indeed a potential solution. But as you say, it's an arbitrary way of doing it (because you're arbitrarily setting the audience weight to 90%).

A possible improvement would be to try to define how the weights are set in a more data-driven way. For example, if there are $N_{u,m}$ user reviews for movie $m$ and $N_{c,m}$ critic reviews for movie $m$, you could create the user-score weight for that movie in the following way: $$ w_{u,m}=\frac{N_{u,m}}{N_{u,m}+N_{c,m}} $$ and the critic-score weight as: $$ w_{c,m}=1-w_{u,m} $$ However, I imagine there might be lots more user reviews than critic reviews. If this is the case, you could start with a sensible assumption such as "every critic review is worth 10 user reviews" or something like that. It will depend on the number of user reviews relative to the number of critic reviews - you can find a multiplier (like e.g. 10) that is sensible for your data.

Another completely different approach would be to do PCA on the two variables and see if you can use the first principal component as your final score. This might make sense since I imagine that the user scores and the critic scores are somewhat positively correlated. This also has the advantage of removing the subjectivity / arbitrariness component.

$\endgroup$
2
  • $\begingroup$ Would it be possible to do PCA even if I don't know the standard deviation of the audience ratings data set? My (limited) understanding is that standardization is the first step of PCA and for that I'd need to know the standard deviation of both data sets. $\endgroup$ Oct 7, 2021 at 1:17
  • $\begingroup$ @GaussianBlur You don't necessarily need to standardise when doing PCA, sometimes it's completely fine (or even recommended) to use the covariance matrix rather than the correlation matrix - it will depend on how different the scale and variance of your two variables are. Can you tell us more about the format of your dataset? $\endgroup$
    – Adrià Luz
    Oct 7, 2021 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.