I am facing the following situation. I am running a web service for which I want to do A/B testing. The two versions A and B differ in the formatting/ordering/content of the web service answer to a user request.
The criterion I compute is the click-through rate in each version, that is to say the number of times users have clicked on the request results over the number of requests made.
However, the main issue is that I do not have access to any end-user/cookie id and thus cannot map directly users to versions (A or B). I can only map requests (randomly or based on their features) to versions.
This means that the assumption that end users assignment to versions is perfectly randomized, traditionally used in A/B testing framework, is not verified here. Furthermore, this means that some requests in version A are dependent on some others in version B and reciprocally.
- Is it meaningful to use bootstrap confidence intervals to compute confidence intervals of the difference between click-through rates?
- Provided I cannot identify end-users, how could I change my experiment setting to improve it and make it easier to interpret?
My first guesses:
I cannot compute any asymptotical confidence interval using Central Limit Theorem as the independence between each request is not validated (2 requests may come from the same user and be dependent).
I cannot use bootstrapping because it uses the fact that both versions are independent when re-sampling them independently...