How would YOU compute IMDB movie rating?

Question

I'm doing this only for learning purposes. I've no intentions of reversing the methods of IMDB.

I asked myself I owned IMDB or similar website. How would I compute the movie rating?

All I can think of is Arithmetic Mean

For a movie data provided below computation would be

(38591*10 + 27994*9 + 32732*8 + 17864*7 + 7361*6 + 2965*5 + 1562*4 + 1073*3 + 891*2 + 3401*1) / 134434 = 8.17055953

---

My rating 8.17055953 which is close to IMDB's Arithmetic Mean.

1. Whats wrong with my score? Why is it not ideal (because IMDB didn't use it)?
2. If you had to compute. How would you have done it? What factors you would consider?

Note:

I'm not asking about the ranking mechanism for the Top/Bottom listing (that is shown here : http://www.imdb.com/chart/top). This question is about how IMDb turns the arithmetic mean into the weighted mean. Take "Manos: The Hands of Fate" — Unweighted mean 2.1, weighted mean 1.5. (Or "The Godfather", another example of ballot stuffing.)

Answer 1

First, define the theoretical construct of interest. There are many ways that a rating can be defined:

• What is the theoretical target population? The entire world, English speakers, people who visit IMDB, people who have seen the movie in question?
• What is the target time frame? It is the rating of the movie now or averaged over its release time.
• Is it a democratic rating or an expert rating? Some people are more knowledgeable about the worth of movies. Some people are better able to differentiate a good from a bad movie. Some people are more consistent in their ratings over time. Should ratings from people who are "better" at rating movies be given more worth. This relates to a philosophical question of aesthetics and the meaning of intersubjective goodness.
• Assuming you could get honest ratings from the entire target population in the entire time frame weighted or not by expertise, what is the mapping between these ratings and the composite rating? This could be the arithmetic mean. Alternatively, there are many other ways of combining individual ratings. For example, you could use an interpolated median. Some alternatives would have minimal effect on the rank order of films, but would have a major influence on the absolute value of the rating.
• Is the number of people interested in the movie relevant to the rating?

Second, use all the available information to estimate the theoretical construct. This is where the issues discussed by others would be important.

• the role of demographic adjustments would depend on your definition of the target population
• a weight for trust could be incorporated. Many indicators could be used:
  • the number of previous ratings (more ratings would suggest someone who is engaged in the site more)
  • the degree to which previous ratings are consistent with other raters or at least a subset of raters (greater consistency would suggest thoughtful and honest responding;
  • the degree to which responses are distributed over an extended period of time (this would suggest that the person is less likely to be attempting to game the system)
  • degree of engagement with the site in general: e.g., accessing the site, contributing to discussion boards (more engagement, more trust)
• as mentioned by @csgillespie you could weight more recent votes greater if you wanted to estimate current attitudes to the film
• You could weight for expertise in ratings. This would be correlated with trust ratings, but there is a difference.

Third, validate and monitor the estimation process using external trusted data sources.

Answer 2

Partial answer. See the help page entitled: The vote average for film "X" should be Y! Why are you displaying another rating?

In short, IMDb uses:

a complex voter weighting system to make sure that the final rating is representative of the general voting population and not subject to over influence from individuals who are not regular participants in the poll.

Also note that:

In order to avoid leaving the scheme open to abuse, [IMDb does not] disclose the exact methods used.

Answer 3

Whats wrong with my score? Why is it not ideal (because IMDB didn't use it)?

If the score was only for your use, then nothing is wrong with your calculation. However, IMDB try to make it difficult for people to obviously influence the final score.

If you had to compute. How would you have done it? What factors you would consider?

Here are a few factors that you could consider (but will be unable to check):

1. The final score may be weighted according to how many votes have been cast.
2. Votes may be weighted by a time variable. For example, votes cast last year are less important than votes cast today.
3. Votes cast by users who have voted for other movies have more weight, i,e. a reputation coefficient.
4. Perhaps they incorporate data from other sites.