I'm attempting to write some code for item based collaborative filtering for product recommendations. The input has buyers as rows and products as columns, with a simple 0/1 flag to indicate whether or not a buyer has bought an item. The output is a list similar items for a given purchased, ranked by cosine similarities.

I am attempting to measure the accuracy of a few different implementations, but I am not sure of the best approach. Most of the literature I find mentions using some form of mean square error, but this really seems more applicable when your collaborative filtering algorithm predicts a rating (e.g. 4 out of 5 stars) instead of recommending which items a user will purchase.

One approach I was considering was as follows...

  • split data into training/holdout sets, train on training data
  • For each item (A) in the set, select data from the holdout set where users bought A
  • Determine which percentage of A-buyers bought one of the top 3 recommendations for A-buyers

The above seems kind of arbitrary, but I think it could be useful for comparing two different algorithms when trained on the same data.

  • $\begingroup$ You have what is called "implicit feedback." If you search for evaluation metrics for implicit feedback recommender systems, you will probably have more luck. Some example metrics include precision, recall, NDCG, and utility/R-score. $\endgroup$
    – Trey
    Commented Mar 3, 2015 at 0:18
  • $\begingroup$ Thanks, Trey. Hopefully I can more precisely hone in on the answer I need using that terminology. $\endgroup$
    – neelshiv
    Commented Mar 3, 2015 at 13:31

1 Answer 1


Recommenderlab is an R package that has built-in functionality to train and evaluate item-item CF based on binary user-item matrix. If nothing else, reading the package vignette might give some ideas. The author of the package also has a paper on evaluating top-n recommendations based on binary user-item data. Links below. The pdf's have the same name but they are diff content.




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.