Offline evaluation of Counter factual data for Recommendation

I am building new model and facing at offline evaluation tasks.

My goal is to predict higher CTR(=click/impression) advertisement, and improve sales.(sales would improve if user watch more attractive ads and click more than before)
At this moment, I have log data which is biased by current running rule; high estimated CTR ads appeared in the logs more often than the low estimated CTR ads.

user_id | ad_id | click_flg
111 | A     | 1
111 | B     | 0
111 | C     | 1
111 | B     | 0
222 | A     | 1
222 | C     | 0


How can I evaluate new model to current running rule in terms of sales without A/B test???

As far as I search, It seems that I can apply counter factual frame works.

Also, I am wondering if I can use replay method of bandit algorithm with counter factual. I am still reading paper, but Risk Minimization for this tasks exists. I would appreciate for any help such as comment, link. Cheers.

You can use inverse propensity scoring (IPS) to evaluate new models if you have access to the probability each particular ad was shown. The idea is to weight the reward by something that controls for the bias in your ad-display process. Let's say your data looks like

$$(x_1, y_1, \delta_1), \ldots, (x_n, y_n, \delta_n)$$

where $$x_i$$ is the context (user information), $$y_i$$ is the action (the ad shown) and $$\delta_i$$ is the reward (the binary click value). Denote the model that you gathered the data under by $$\pi_0$$, so that $$\pi_0(y|x)$$ is the probability you show ad $$y$$ given context $$x$$.

Now, to test a new model $$\pi$$, the IPS estimator is given by

$$\hat{U}_{ips}(\pi) = \frac1n\sum_{i=1}^n \delta_i \frac{\pi(y_i|x_i)}{\pi_0(y_i|x_i)}$$

This will give you an unbiased estimate of expected reward for your new model based on data gathered under the old model. In other words, this is the reward you expect to have gotten if you had deployed $$\pi$$ instead of $$\pi_0$$.

The reason why this works is that the weight adjusts for the sampling bias in $$\pi_0$$. For example, suppose an observation is unlikely in the old model and likely in the new model, then the weight $$\frac{\pi(y_i|x_i)}{\pi_0(y_i|x_i)}$$ will be large. This makes sense because we probably only have a few reward samples for this observation under the old model, while we would expect to see it more frequently in the new model, so we should weight it higher. The same logic works in the reverse, relatively less frequent observations in the new model will result in a lower weight.

I would suggest this SIGIR 2016 tutorial by Thorsten Joachims for more information on counterfactual evalution.

• thank you for your help. That tutorial helped me a lot as well. Can I ask a few more question? 1. this method can be applied to other model base prediction, not bandit?? I might use logistic regression to predict CTR and recommend highest CTR ads for each past logs, and calculate ips estimator. Is this ok? I am just afraid that ips estimator stuff can be limited only for bandit algorithm. Commented Feb 21, 2019 at 1:24
• 2. I do not need to estimate propensity score all the time, right?? If assignment mechanism is under our control, I do not need to estimate propensity score and use system assignment probability, according to "Experimental vs Observational" section in "Counterfactual Evaluation - I" . I am just afraid that no need for estimating propensity score sound unfamiliar cuz I have been studying counterfactual field and have never experienced that. Commented Feb 21, 2019 at 1:31
• 1. Yes you can use logistic regression, or anything else really. 2. Yes. If you control the assignment mechanism, then you already have the propensities and can directly calculate IPS estimator. If you don't control the assignment mechanism then you must estimate the propensities first. Commented Feb 21, 2019 at 2:45