I am currently trying to verify and evaluate a personalized recommender system I am working on, which seems like a huge task. Evaluating a static recommender system is rather easy and can be done with a simple bayesian framework the fast way.

But my problem lies in how to find a good way to measure the degree and quality of personalization without too much user feedback. User interaction would be OK, but the whole thing should work without the user giving much explicit feedback.

Are there any good meta-strategies to tackle this problem?

What are good values I should measure? At the moment I think about values that do not directly depend on a conscious user feedback (i.e. "satisfaction"), but on more statistically inferable factors like convergence factor of the learning algorithm, etc.

Hints anyone? Thanks guys. I don't need a full strategy, I am more hoping that anyone already found good resources how to deal with a variety of these problems, then I may just look what suits best for my specific case.

Best Martin

  • $\begingroup$ Do you know when a user uses your recommendation (e.g., rents a recommended movie)? $\endgroup$
    – cyborg
    Commented Nov 18, 2011 at 17:00
  • $\begingroup$ If your system uses real data to train the recommendation algorithm then you could use some part of the data as a hold-out set and have the system predict the choices in the holdout after training. As implicitly suggested by @cyborg above, once the system goes live you could use actual choices (if any) from your recommended set to evaluate the quality of the system. $\endgroup$
    – varty
    Commented Nov 18, 2011 at 17:06
  • $\begingroup$ @cyborg: Yes, I will know when a user uses the recommendation once the system is online - but it isn't so far. $\endgroup$
    – mare
    Commented Nov 19, 2011 at 10:43
  • $\begingroup$ @varty Totally right to do it that way. But before it is live, this still needs to somebody judge subjectively on how good the personalization is so far.. Which was what I wanted to avoid. $\endgroup$
    – mare
    Commented Nov 19, 2011 at 10:47
  • $\begingroup$ I think the most important part is how to measure the personalization itself - hot to measure a static recommendation is already well-known. $\endgroup$
    – mare
    Commented Nov 19, 2011 at 14:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.