Statistical significance of item-to-item relationship

Question

Context: I have an e-commerce application - so I have users and products. I'm trying to build an item-to-item recommendation system based upon user behavior. In particular I'm taking all the users' product view histories and for each product I'm listing out the products that cooccur with it and tallying up number of times that each one cooccurs.

So if we have the following user product views:

• user 1 viewed A, B, C,
• user 2 viewed A, B, D,
• user 3 viewed A, C, E,

Then this would be to cooccurrence counts for each product:

• A cooccurred with B twice, and C twice, and E once
• B cooccurred with A twice, and C once, and D once
• C cooccurred with A twice, and B once, and E once

But the question I can't solve is "How significant is the number of times that A occurred with B? Can it be attributed to simple luck?" In particular, if A and B are both unpopular products but nevertheless A cooccurs often with B, then this is significant. However if A and B are both popular, then perhaps the cooccurrence can be attributed to chance.

How can I establish a threshold for statistically significant cooccurrence between each pair of products?

(see this related question)

Answer 1

I have an idea. First, I know these things:

• I know how many items there are.
• I know how many times each item has been viewed. (say, over the past month)
• I know how many coocurrance "pairs" there are. For instance in the example above user 1 viewed A, B and C - this is 3 pairs: AB, AC, and BC.

Given this then I can infer some things:

• Under an assumption that individuals view items based solely upon popularity, then the probability that any particular item gets viewed is $P(\textrm{view ItemX}) = \frac{\textrm{number of times ItemX was viewed in the original data set}}{\textrm{total number of views in the original data set}}$
• Also the chance for a pair occurring between ItemA and ItemB is $P(\textrm{ItemA and ItemB}) \approx P(\textrm{view ItemA})\times P(\textrm{view ItemB})$

The probability distribution of how many times ItemA and ItemB cooccur is then a Binomial distribution with $n=$total number of pairs in the original data set and $p=P(\textrm{ItemA and ItemB})$. This is all I need for a statistical test.