1
$\begingroup$

I implemented a recommender using tensorflow, based on e-commerce data. This recommender is predicting the next item a user will buy. I will judge the performance of my fitted model, by getting the next estimate of the model and comparing, if the user did actually buy this item (compare precision_at_k with k == 1).

My precision is around 20% (meaning in 20% of the cases the user did actually buy the product the system recommended). Our shop has about 150 Products and my evaluation is based only on properties of the user, not his/her purchase history.

In this case, is 20% good? Are there other, similar systems I can compare to? What would be a good precision for, e.g. 100.000 products?

$\endgroup$
6
  • $\begingroup$ @Sycorax Thank you very much for the hin. You are right, the language and framework are not really important here. Do you have an answer to the question, though $\endgroup$ Commented Jun 18, 2018 at 15:46
  • 2
    $\begingroup$ This doesn't really seem answerable in a context-free setting. Is your model good enough for what purpose? You could conduct a literature review and find publications claiming to obtain some level of precision using some model & compare your results. Or, if you're trying to make money using recommendations, then you can set a certain target for sales. Or, if you're doing this for a class, you need to make sure this project satisfies the rubric. I don't think anyone can answer these questions for you. $\endgroup$
    – Sycorax
    Commented Jun 18, 2018 at 15:48
  • $\begingroup$ @Sycorax So far I did not find recent literature, which evaluates the accuracy of recommender systems. All I am interested in is how does the accuracy of this model compare to e.g. Amazon, Netflix or anyone else. I really have no idea, if 20% are awesome or terrible $\endgroup$ Commented Jun 18, 2018 at 15:52
  • $\begingroup$ This is an article about Netflix's recommender system. dl.acm.org/citation.cfm?id=2843948 Looks promising. $\endgroup$
    – Sycorax
    Commented Jun 18, 2018 at 15:54
  • 1
    $\begingroup$ I think is allways good to try with some benchamrks. Such as 1) reomend allways the most popular item, 2) recomend the last viewed item of that user. This way you can check which precision those benchmarks have and by how much your recomender is imporving it. $\endgroup$
    – PolBM
    Commented Jun 18, 2018 at 16:06

2 Answers 2

3
$\begingroup$

An easy and widely applicable way to help interpret the performance of a model is to compare it to an appropriate baseline. For example, imagine a trivial model that predicts that everybody will buy the single most popular product every time. What is its performance under this metric? If its performance is close to or better than your model of interest, then your model isn't accurate enough to be useful.

$\endgroup$
2
  • $\begingroup$ That sounds somewhat similar to an A/B-test. Comparing my model against a random recommendation. Is that correct? $\endgroup$ Commented Jun 18, 2018 at 16:28
  • $\begingroup$ @User12547645 No, "A/B test" is another term for a randomized experiment, which involves experimental manipulations on real subjects, not just data analysis. Furthermore, I'm suggesting predicting the most popular product every time, not a random product. $\endgroup$ Commented Jun 18, 2018 at 17:23
2
$\begingroup$

As others already noticed, to answer the question "how good" is your recommender, you would need some kind of benchmark. How did you recommend the products before? If you had some kind of recommender system, use it as your benchmark. If you didn't, there are several possible choices for benchmarks:

  1. randomly recommended products,
  2. recommending the same most popular product (especially if the distribution of product purchases is very skew),
  3. the last product bought by each user as a recommendation.

Each of the scenarios can serve as a benchmark of a naive recommender system, if you can do better then those scenarios, then your recommender is worth considering. Notice that such tests can be partly done with dry run: simply make your recommendations based on data until month $t$ and assign "recommendation" scores for products in month $t+1$ (in the past), so to check what did the users buy at $t+1$ time. This doesn't account for your marketing efforts (you didn't send or show them the recommendations!), so this wouldn't give you final answer, but it is a good starting point.

If I were you, I'd also think of using different metric for assessing the recommender performance. The two alternative choices for measuring performance of recommender systems with implicit ratings (users did not state preferences explicitly), are Mean Percentage Ranking (MPR) and Mean Reciprocal Rank (MRR).

The first metric is defined in terms of percentile ranks, where $rank_{ui}$ is rank of $i$-th product for $u$-th user, where the ranks range from $0\%$ (most preferred) to $100\%$ (least preferred), and in terms of indicator function $d_{ui}$ that is equal to $1$ is $i$-th product was bought (clicked etc.) by $u$-th user and $0$ otherwise,

$$ MPR = \frac{\sum_{u,i} d_{ui} \times rank_{ui}}{\sum_{u,i} d_{ui}} $$

as you can see, the measure is simply mean of the percentile ranks that were bought by the users. $MPR=0\%$ means that your recommender is perfect, $MPR=50\%$ is the score that you'd see for recommending random products and $MPR=100\%$ is the score you'd achieve if all your recommendations were wrong. This score has very clear interpretation, so it may be useful. Moreover, notice that if you have $k$ products, then $MPR \times k$ is the average position in ranking of the products that were bought, so if $MPR \times k = 3$ and you recommend $4$ product for each user, then you should be happy with the results.

The second measure is defined in terms of ranks $rank_{ui}$, where $1$ is the most recommended product and $k$ is least recommended product,

$$ MRR = \frac{1}{n} \sum_u \min_{i \in \{ i \,\mid\, purchase_{ui} > 0\}}(\,rank_{ui}\,)^{-1} $$

The score is calculated by taking average of the lowest ranks of the product that was bought by users. The higher score the better, where $MRR=1$ means perfect matching. This score has is defined in terms of harmonic mean, so $\tfrac{1}{MRR}$ gives you the position of the product with lowest rank that was bought.

I find those two metrics more intuitive and easier to interpret the precision at $k$ and would recommend trying them.

I wouldn't agree with the advice that you could compare your recommender to Amazon, Netflix, Spotify etc. This generally doesn't make much sense unless you have similar product and users (I guess you don't). If you have different products and different users, then there is no reason to expect similar performance scores. Notice that even if you had same products and users, then still your platform may differ, you may have different marketing strategies and tools, different prices, availability etc., so lots of factors may influence your performance.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.