My boss wants us to run a simple experiment where we manipulate the search results on our website. He is insisting that we must have counterfactuals for the results to be valid. The way I see counterfactuals is as a way to improve statistical power and more rigorously evaluate causality, so I'm not sure why the absence of them would make an experiment invalid. Im wondering if anyone could share their thoughts on this issue, as I think I'm missing something.
Counterfactuals to me means comparing what happens with the intervention (here doing something to search results) to what would have happened without the intervention. Usually, you can't see that for the same user & search & done at the same time, because the same user may not even ever do the same search again and if they repeat it later its complicated (circumstances change, initial search results may still influence the user on the second occasion etc.). So, what you do is randomly assign whether the intervention is applied to a user or not, and then you compare what you're interested in for their next search. Of course, you have to be careful to not count multiple searches by the same user as if they had been completely new search by completely different users - if you want to look at multiple searches and/or sessions per user stuff is a little more complicated - if you can't keep your users identified e.g. due to data privacy that's also tricky. There's also worries about learning effects (e.g. if you change the way results are shown, maybe people need to get used to it first, which would be a trickier thing to evaluate properly and would require looking at multiple searches, but also to take into account that some users may not do multiple searches within your time horizon etc.).
As long as it is the nice simple case of one search with or without intervention per user and assignment is random (simple A/B test), it's kind of straightforward to compare these (assuming the outcome is defined for everyone etc.). If you don't use such a setup and e.g. do searches on some days for everyone without intervention and for everyone with, then that's pretty problematic. Perhaps you accidentally compare whether searches result in sales during the week before black friday (people perhaps more likely just looking without buying) vs. the black friday week (more likely to buy) etc. and perhaps it's not that obvious, but there's still some kind of biases that you just don't know about (perhaps people are more worried about the economy and don't want to spend money because of something in the news etc.). Only a proper randomized A/B test helps there.
If you have additional information that could of course help make the comparison more efficient (e.g. stratification or covariate adjustment for previous user behaviour - e.g. previous sales per search, clickthrough rate etc., it's just often important to put this on a good scale such as per search or per website visit etc.).