2
$\begingroup$

I have some trouble understanding the concept of using precision and recall to evaluate a content-based recommender.

Suppose I want to recommend articles to users. A content-based recommender will first get my article reading history, then find articles similar to what I have read based on some clustering result.

Here's a graph about precision and recall:

enter image description here

If I understand correctly, the true positives part is fixed, which is always the number of articles I've read. That means the the false negatives part is 0, thus recall is always 1?

$\endgroup$
3
  • $\begingroup$ It depends on your definition of true positive (i.e. a successful guess), hence your performance measure. this and this might be related. $\endgroup$
    – jeff
    Jun 16, 2016 at 22:18
  • $\begingroup$ @HalilPazarlama I guess a more appropriate definition for precision and recall for recommender systems would be, precision = (# of recommendations in the user's library)/(total number of recommendations), recall = (# of recommendations in the user's library)/(# of articles in the library). $\endgroup$
    – Roy C
    Jun 16, 2016 at 22:24
  • $\begingroup$ I see. then the number of false negatives should be the ones you returned but not in the user's library, which is not necessarily 0. right? $\endgroup$
    – jeff
    Jun 16, 2016 at 23:20

3 Answers 3

0
$\begingroup$

Sounds like you don't have labeled data. If I understand your problem correctly, a false negative would be an article that you predict would be "interesting", but that you have not yet read. If this is correct, then you would want to maximize your false negatives!

Read up on ways to evaluate unsupervised learning

Googled and found http://research.microsoft.com/pubs/115396/EvaluationMetrics.TR.pdf

$\endgroup$
0
$\begingroup$

Paying attention to the meaning of the words "precision" and "recall" makes it considerably easier to understand them. Recall simply means how many of the points that are positive you recall correctly, i.e. the proportion of positives that you actually lable as positive.

Precision is simply the precision of your predictions, i.e. if you predict $n$ times that something has a positive lable, the precision is the proportion of making a correct decision when labelling something as positive

$\endgroup$
0
$\begingroup$

The usual scenario for a recommender system would be that for a particular user it scores the items in terms of how likely are they to be preferred by the user (clicked, watched, listened, purchased, etc). You do this by training your algorithm on the items the user interacted with (watching history on Netflix) to score the items that the user didn't interact with. To test the model, you would usually hold out some items per user (pretend to your model that the user didn't watch this movie) and ask the model for predictions. The recommender system metrics would judge if your model gives high scores to the items user actually has preferred and low scores for the items they didn't. To to this, there are specialized metrics, but you can also use metrics such as Recall@$k$: recall calculated when marking the $k$ items with highest scores as "positive" prediction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.