# Affinity analysis dealing with power law distribution

How do you account for top selling/most central items when analyzing affinities? For example, let's say you're analyzing affinities among movies based on people who rented both. How do you deal with the fact that virtually everyone went to see a blockbuster and the presence of many people who rented Lord of the Rings and Lars and the Real Girl does not constitute a tie?

Another way of asking is why doesn't every suggested item on Amazon's products' page include socks. Everyone buys socks.

• "Everyone buys socks." That's rather culture-bound. Jul 4, 2013 at 14:24

The way these systems (called recommendation engines) usually work is with item-based filtering. There are various places to read about this. There is a good tutorial in Chapter 2 of Segaran's Programming Collective Intelligence or on Stackexchange (for example this question.) But basically, the problem you have suggested doesn't seem to be a problem because of the way similarities (or affinities) between items are calculated.

The basic idea is like this: suppose you want to recommend items to users based on whether other users bought the same item. You have a list of items, call them $A, B, C, \ldots$, which users might buy and a list of users $a, b, c, \ldots$ who each bought some of the items. You make a large table like this

  A B C
-------
a 1 0 0
b 1 0 1
c 0 1 0


with a $1$ if user $x$ bought item $Y$ and a $0$ otherwise. For each item, you get a vector, which basically records which users bought it: $A$ would correspond to $(1,1,0)$, $B$ to $(0,0,1)$ and $C$ to $(0,1,0)$ in this example. You then compare items based on how similar their vectors are, perhaps by counting the places where they differ. In this example, suppose someone had bought $A$ and we wanted to tell them what they should buy next. $A$ differs from $B$ in $3$ places and from $C$ in one place, so we should recommend $C$.

It is also possible (and better) to use more complicated notions of distance between vectors and to take into account how users might have scored items etc, but this seems to be the general idea.

Coming back to your question, if an item had been bought by everybody, its vector would be $(1,1,1,\ldots)$. It is quite possible that this might not be close to anything and so not get recommended. In the above example, suppose there is a new item $D$ which has been bought by everybody. Suppose a user buys item $C$. The distance of $C$ from $A$ is $1$, the distance of $C$ from $B$ is $2$, and the distance of $C$ from $D$ is $2$, so this user would have $A$ recommended to them instead of the mega-popular $D$.

In your scenario, $C$ could be Lars and the Real Girl, $A$ might be Zack and Miri make a Porno, $B$ might be some other fairly niche movie and $D$ might be Lord of the Rings. Someone who had rented only Lars would have Zack recommended to them rather than Lord of the Rings.

• But let's say that some products are vastly more popular than others. So, a little known movie comes out that only tree people have seen, so it has 0s in all of the other columns, not because the other people didn't choose it, but because they haven't been exposed to it. By contrast, everyone has seen Lord of the Rings, and just because someone watched it doesn't mean that they're "the type of person" who likes fantasy adventures based on books. They just watched because it was popular. How could you account for this to recommend the new item, instead of the blockbuster? May 31, 2013 at 21:57
• Items don't correspond to rows in the table; they correspond to columns. In this recommendation scheme, when someone buys an item, you recommend whatever item's column is closest to the column of the item they bought. The idea is to look at the whole thing from the point of view of the items, not from the point of view of the people. May 31, 2013 at 22:16

The notion that someone watched 'Lord of the Rings' because it was popular, and not because he likes those kind of movies is very relevant to the topic of recommender systems. It shows that there are several other facets of recommendation, than just the interest-type of the user. Some of them are:

• Popularity - as you mentioned, someone might watch a movie because a lot of people have watched it.
• Freshness - occasionally, a user might want to try out a different type of movie than those he watches regularly.
• Diversity - while recommending a set of movies to watch to a user, it makes sense to give him choices which are slightly different from each other, for example, suggesting 'Lord of the Rings: Part One', 'Stardust', and 'Harry Potter: Part One', might be more useful to the user to decide what to watch than suggesting the three 'Lord of the Rings' movies.

One possible way of dealing with popular items specifically is this. To start with, popular items will be incorporated in everyone's interest profile. This is usually done by subtracting the movie rating average from all ratings entries as a preprocessing step to the modelling. Hence if 'Lord of the Rings' has an average rating of 4/5, the predicted rating for a 'neutral' user will be 4**. Only if he is particularly averse to fantasy, the predicted rating will reduce. This goes by the logic: If I don't know anything about a user, I might as well suggest 'Lord of the Rings', since it's so popular.

Once in a while, the system will try out 'fresh' recommendations to users - these might be new movies, or genres the user doesn't usually watch, so as to discover changes of the user's tastes. The system will also monitor the user's response to the recommendations. If a user repeatedly ignores a particular recommendation, then the system will reduce the importance of that recommendation, in effect, incorporating the user feedback to the recommendations. The system can thus 'learn' about the user better. Maybe this particular user doesn't want to watch 'Lord of the Rings' or maybe he has already watched it. In your example, however much 'sense' it makes to show socks ads to people, if they repeatedly ignore it for some reason, it is better to stop showing them.

** For simplicity, I have assumed that popular items have a high average rating, but this need not be so. In general the preprocessing can proceed in a similar way with some modifications.