Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

I am currently trying to verify and evaluate a personalized recommender system I am working on, which seems like a huge task. Evaluating a static recommender system is rather easy and can be done with a simple bayesian framework the fast way.

But my problem lies in how to find a good way to measure the degree and quality of personalization without too much user feedback. User interaction would be OK, but the whole thing should work without the user giving much explicit feedback.

Are there any good meta-strategies to tackle this problem?

What are good values I should measure? At the moment I think about values that do not directly depend on a conscious user feedback (i.e. "satisfaction"), but on more statistically inferable factors like convergence factor of the learning algorithm, etc.

Hints anyone? Thanks guys. I don't need a full strategy, I am more hoping that anyone already found good resources how to deal with a variety of these problems, then I may just look what suits best for my specific case.

Best Martin

share|improve this question
Do you know when a user uses your recommendation (e.g., rents a recommended movie)? – cyborg Nov 18 '11 at 17:00
If your system uses real data to train the recommendation algorithm then you could use some part of the data as a hold-out set and have the system predict the choices in the holdout after training. As implicitly suggested by @cyborg above, once the system goes live you could use actual choices (if any) from your recommended set to evaluate the quality of the system. – varty Nov 18 '11 at 17:06
@cyborg: Yes, I will know when a user uses the recommendation once the system is online - but it isn't so far. – mare Nov 19 '11 at 10:43
@varty Totally right to do it that way. But before it is live, this still needs to somebody judge subjectively on how good the personalization is so far.. Which was what I wanted to avoid. – mare Nov 19 '11 at 10:47
I think the most important part is how to measure the personalization itself - hot to measure a static recommendation is already well-known. – mare Nov 19 '11 at 14:06
show 1 more comment

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.