# How to check if two segments of an A/B test have significant difference in the treatment effect

I am planning to launch an A/B test and there's a concern about a possible primacy/novelty effect. I've decided to segment users into two segments (New Users, Returning Users) and estimate if the effect of the treatment significantly different between the segments. The logic is following: if I detect the difference between these two segments, this should alert me to the fact that returning users react differently to the treatment that the new users, hence, possible primacy/novelty effect.

So far I've read about Conditional Average Treatment Effect and Heterogeneous Treatment Effect but it's just words for me and I can't figure out how to apply the respective methodology to my problem.

Could anyone point me in the right direction?

• You might just use a time covariate and test for sig arxiv.org/pdf/2102.12893.pdf Commented Oct 4, 2023 at 14:30
• This whitepaper is actually the only source I have found that talks about detecting presence of primacy/novelty effects (well, there's another one from google employees that this one mentions). But I have a few problems with applying the methodology: 1. in my setting, the metric of interest is a conversion rate and the paper only mentions the case where the user repeats their action. 2. there's no detailed example of applying the described methodology anyways so I'm having a problem with that too. If you could help me with the real example - that would be an immense help. Commented Oct 4, 2023 at 17:05

I would fit a linear regression model for the expected conversion rate in the four groups of users (treated new, treated returning, control new, and control returning) like this:

$$E[Y \vert Returning, Treated] = \alpha +\gamma \cdot Returning + \beta \cdot Treated +\delta \cdot Returning \times Treated$$

with heteroskedasticity-robust standard errors.

This setup assumes that each observation is a distinct user. Binary $$Y$$ indicates that they converted during the test period. $$Returning$$ means that this user has some experience with the site/app from before the test starts.

The effect of treatment for new users is $$\beta$$. For returning users, it is $$\beta + \delta$$. The null hypothesis that $$\delta = 0$$ is a test of primacy/novelty effects. Primacy usually means that $$\delta < 0$$. Novelty means that $$\delta > 0$$. You could also use one-sided hypotheses here if you are interested in distinguishing between them.

People balk at using a het-robust linear probability model (LPM) rather than a logit/probit, but in a fully saturated model with robust SEs, the estimates of the average marginal effects will be identical, but the effects are easier to interpret with the LPM. Interactions can get tricky in nonlinear models.

Kohavi, Tang, and Xu (2020) has some nice practical references on novelty/primacy, though the book itself does not get as technical as what I wrote above.

Kohavi, Ron, Diane Tang, and Ya Xu. 2020. ​Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. https://experimentguide.com/

• Thanks! I've read the book and that's where I got a reference to this: egap.org/resource/… Any resources that I've looked at are way too technical and lack solid examples though. Even though the mentioned resources seemingly provides the guidance, it's too hard for me to find a way to apply it without any examples. Commented Oct 4, 2023 at 19:51
• @dimitriy It seems like we might need to plot our metric against time after first user visit. then, we look for a steady point such that we find no significant difference in terms of the interaction with time? Then, we filter out earlier values. Thoughts? Commented Oct 4, 2023 at 20:01
• I am not sure if that is doable, given the use patterns at @EugeneKrall's site that he described in the comments on the question. I have seen that approach succeed with tools like email clients or news sites, where usage is more frequent, so one can see how the effect changes as exposure to treatment grows. Commented Oct 4, 2023 at 21:55
• My thoughts were, I don't strictly need to apply methodology designed for estimating learning effects pers se. If I already have two segments - new users who received treatment and old users who received treatment. Theoretically I only need to estimate if there's difference in treatment effect between these two groups and I'm thinking maybe Kolmogorov-Smirnov Test or even Chi-Squared Test for Homogeneity? Commented Oct 5, 2023 at 8:55
• I am not sure that says anything about treatment effectiveness. That simply compares new and returning users who are treated. It would be best if you compared the effect for new vs. returning users. The second benefit of estimation is that you might have a situation where the primacy/novelty effect is stat stig but not economically important. Having an estimate helps you decide if that matters. Commented Oct 5, 2023 at 18:37