I developed a market basket program for a retailer. I used the pymining library association rules function. I assume this uses the Apriori algorithm. I got 340,000 rules for an input of 17,000 invoices each with a list of corresponding items.
My colleague developed a R program using Jaccard coefficient for the same set of 17,000 invoice items. The R program gave a set of 9000 similarity coefficients for combinations of 2 items. The store has 350 active items. This algorithm assumes that if 2 items were bought in the same basket, they were 'similar'. This had to be done because there is no rating available in this store.
I then displayed both our results in a common grid. The columns were
Items in basket,
items recommended by pymining,
items recommended by Jaccard coefficient.
So for example, if I had items [1, 2, 3] in the basket, and [4, 5] in pymining recommended, I picked the items with highest similarity for [1,2,3] from the R output, and got say [4,7,8].
What I find is that there is very little correspondence between the recommendations of the market basket and the Jaccard coefficient.
- Is this expected behavior?
- Can we expect the outputs of these 2 algorithms to give similar outputs?