0
$\begingroup$

The company whose data I am working with is not willing to or ready to do A/B testing to evaluate my system's accuracy. They are asking me to come up with some other way to get some preliminary results, and if they turn up positive they are willing to do the A/B test. So, how do I do it?

Let me give you an example from my real data. There are 21 genres of movies, and a certain user watches 1 movie of genre 13, and 17 movies of genre 14, shown below.

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 17, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0

My system tells to recommend genre 11 next. So how do I know how good is my recommendation accuracy without first recommending the user some movies of genre 11 and checking whether and how many of them he/she opens?

$\endgroup$
  • 1
    $\begingroup$ You can't check performance without data. You need data some how. What I would do in your situation is (I am assuming the company you are working with has historic data) if you have a user who has 1 movie of genre 13 and 17 movies of genre 14 at time x then go to time x+k (you choose k to be some reasonable number) then you see which genre of movies they liked most (besides 13 and 14) and see if it matches what your recommendation was. Then do this for many users. $\endgroup$ – user3494047 May 10 '17 at 9:07
3
$\begingroup$

An alternative to what user3494047 suggests, is to hide a few data points at random for every user, make recommendations using your algorithm, and then uncover the hidden data and see how many of those matched the recommendations.

Since you have the actual count of the movies of a particular genre, watched by each user, you can use it as a proxy for the strength of their preference for that genre.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.