1
$\begingroup$

I am looking for the best known way to correlate movie taste, i.e. to determine which people out of a large pool deviate the least from your own movie scores.

One site with this purpose is Criticker. As a longtime user, I have over the years read and participated in many discussions about its algorithm and its shortcomings in a quest to find a better one.

First, I'll briefly explain how the current algorithm works and what its drawbacks are. And then my question would be: Do you know a better one?

current algorithm:

  • Each users scores movies.
  • The user can freely choose a ranking scale anywhere between 0-100 (so 1-10 or 0-5 or 50-100 are all valid).
  • Movies are not compared to each other by their nominal score but by their percentile rank.
  • This is done in order to
    • normalize the ranking scales (which could also be done linearly instead)
    • be able to compare different voting behaviour, i.e. different use of the same ranking scale (see "advantages" below)
  • To compare the similarity of taste between 2 users, the differences between the respective percentile rank of each movie in common are averaged.

example:

movie 1 is in 76th percentile of user A and in 52th percentile of user B: difference=24
movie 2 is in 99th percentile of user A and in 90th percentile of user B: difference=9
movie 3 is in 89th percentile of user A and in 86th percentile of user B: difference=3
average difference = (25+9+3)/3 = 12

In practice an average difference (called "taste compatibility index" or TCI) of 12 is considered a quite good match.

advantages:

  • Users with different voting behaviour can be compared to each other. For example: Movies ranked 70 are considered terrible by user A but excellent by user B (although they both use a 0-100 scale). In theory the use of percentiles should allow for a more meaningful comparison, as movies are not looked at and compared to each other in isolation, but their relative place (upper end, lower end) on a user's voting spectrum is compared to each other.

drawbacks:

  • When users tend to watch only movies they like (which is the main purpose of this site), they end up having movies with very high nominal values in very low percentiles.

  • And as a consequence of this, 2 users with identical taste get a bad TCI match, if one of them has watched more bad movies. Example: User A and user B have 100 films in common with very similar (or even identical) nominal scores. But if in addition to that user B has watched 50 terrible movies (which user A would find just as terrible, but has avoided for obvious reasons), this impacts user B's percentiles in such a way that his shared films are in much higher percentile than for user A (although they nominally have similar or even identical scores).

  • A special case of the above is this: you create user account A and duplicate the account as user B (i.e. with exact same movies and exact same scores). The next day you score a couple of bad movies only with user account B. This skews the percentile distribution and, as a result, you don't match your own taste (user A) anymore, although your taste has not changed.

The question is:
Is there a better method (than just comparing percentiles) that would still offer its advantages (comparing different voting behaviour) but doesn't suffer from the flaws listed above ?

In the site's discussion forum, a user suggested using Spearman's rank correlation coefficient as an improvement (but without further explanation). I have found a specific example of how to use the SRCC for movie taste comparison, but given my humble maths skills I would like to inquire what the improvements (if any) using SRCC would achieve and what disadvantages it has?

And SCRR aside, are there even better methods for the purpose at hand ?

$\endgroup$
17
  • $\begingroup$ Not sure if this would work, but maybe its worth a try: When a new user creates an account i would ask him or her to name the worst movie he or she ever saw and also the best movie. In addition i would ask the person to rate the movies on a desired scale. From now on i would use this two values as "use of scale" parameters for the person and would transform all follow-up ratings withing this range (given rating for worst movie becomes 0, given rating for best movie becomes 100). Before transformation it might be a good idea to update the min/max, just in case the person found new high/lowlight $\endgroup$ Nov 27, 2018 at 16:20
  • $\begingroup$ @TinglTanglBob: The question about the best/worst movie is unnecessary, because this is determined by the highest/lowest score. But other than that: What you describe is exactly what I meant by "which could also be done linearly instead". It's not really relevant to the question though, because normalizing/equalizing different voting scales is no problem at all and can easily be done linearly (e.g. a 0-5 voting scale becomes 0-100 by multiplication of 20) as you described yourself. The problem is not different voting scales. The problem is different use of the same voting scale (item 4b) $\endgroup$
    – summerrain
    Nov 27, 2018 at 16:28
  • $\begingroup$ Well, as you've said a user could only watch and rate movies he/she likes. What if a users lowest score is 70. Does that mean he watched filmes he liked a lot only or does it mean he dishes out 70 as a min value, even for movies he really doesn't like. $\endgroup$ Nov 27, 2018 at 16:31
  • 1
    $\begingroup$ I don't trust movie scores. they have too much bias of all kinds in them. for instance, i'm lazy and don't score every movie that i watch etc. it's more reliable to observe watching habits, in my opinion. starting from you watch, but also to account for whether you saw the whole thing, time of week and year, etc. Overall, I find recommendations from all streaming services pretty bad, despite the claim that they take into account my preferences, I don't see it in results. most of what they recommend is garbage to me, which I don't bother even trying. $\endgroup$
    – Aksakal
    Nov 27, 2018 at 16:40
  • $\begingroup$ @TinglTanglBob: Yes,this seems to be the conundrum. Your scenario1 may be an avid genre fan only watching and ranking movies of this genre >70. In your scenario2 the value of 70 just has a different (worse) meaning to the user. If the numbers had the same meaning for all users, it would be better to compare nominal values. The reason for comparing percentiles instead, is that some users (e.g.myself) consider 10 terrible and 50 already quite interesting, while others deem everything below 60 terrible. The question is: Is there another, better method than comparing nominal values or percentiles? $\endgroup$
    – summerrain
    Nov 27, 2018 at 17:04

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.