AB test with interaction between observations in the two groups Suppose I am the owner of an e-commerce store. I'd like to design an AB test to measure the average effect of discounts on sales. I am not able split user traffic so that two users see different discounts for the same product. But I can randomly split the products into two groups and apply different discounts to the two groups.
The problem with the latter approach, I believe, is that I am likely overestimating the effect of discount on sales. By splitting the products into discount groups A and B (say, A with 10% and B with 20%), it's not possible to say that the average effect of a 1% discount increase is
$\frac{Sales_B - Sales_A}{Discount_B - Discount_A}$
because there are people who would have bought from group A instead of B had both groups had the same discount. Put differently, some people would have preferred group A products if they were as cheap as group B.
Is there a way to control for this bias? What's the actual name of it? I understand that the observations are not IID, since some products can interfere with the sales of other products. Could you point me to some other areas where such a problem might appear?
 A: *

*I would say


some people would have preferred group A products if they were as cheap as group B

is a perfectly legit effect of discount on sales. You may also say "some people would have preferred products from your competitor if they were as cheap as group $B$".


*There may be bias in your estimated effect on sales because, by coincidence, you might have chosen products in group $A$ that are more susceptible to discounts than products in group $B$ or vice versa. You should check the effect on sales on every single product, then compute the variance $\sigma_{A, B}^2$ of the effect for both groups. The posterior standard deviation on your estimate on the effect of discount on sales is roughly $\Delta_{A, B} = \frac{\sigma_{A, B}}{\sqrt{N_{A, B}}}$, where $N_{A, B}$ is the number of products in group $A$ or $B$. You may consider doing a t-Test if you see some overlap, e.g., if $(\mu_A + 2\cdot\Delta_A) > (\mu_B - 2\cdot\Delta_B)$, where $\mu_{A, B}$ is the effect on sales (however you measure it).


*You should not assume a linear relation between the discount and the effect on sales, as your equation suggests. Assume you offer 100% discount: probably almost everybody who sees your offer would buy it for no cost.
If you can measure the amount of people who see a product in your e-commerce store and the amount of people who buy/don't buy, I would do a [logistic regression][2] for the conversion rate, i.e., $\frac{number~of~sales}{number~of~product~views}$ to measure the effect of discount. Calling the conversion rate $q$, the model would look like
$$q(d) = \frac{1}{1 + e^{-(w_0 + w_1\cdot d)}}$$
where $d$ is the amount of discount. The likelihood function you need to maximize w.r.t. the model parameters $w_0, w_1$ is
$$\mathcal L = \prod_{n = 1}^N q^{y_n}(d_n; w_0, w_1) \cdot (1 - q(d_n; w_0, w_1))^{1 - y_n}$$,
where $n$ runs over all product views, $d_n$ is the amount of discount for the product of impression $n$, and $y_n = 1$ if impression $n$ ended with a sale and 0 else.
In the end you'd get a function for the conversion rate depending on the discount.
These are my first ideas on this problem. You may as well estimate the uncertainty on the model parameters $w_0, w_1$ if you have only few data, e.g., with a Laplace approximation.
[2]: https://en.wikipedia.org/wiki/Logistic_regression#:~:text=Logistic%20regression%20is%20a%20statistical,a%20form%20of%20binary%20regression).
