I have a webpage with a "purchase" button. Users must be logged in to see the page. I would like to A/B test two versions of the button. Each user will consistently see one version of the button for the duration of the experiment. That is, the unit of diversion is the user.

Should I measure the click-through probability of the button (count of distinct users who clicked the button/count of distinct users who saw the button), rather than the click-through rate (number of clicks/number of impressions)?


1 Answer 1


You can do better than either, which is to model at the level of impressions but allow the model to account for per-user variability. Consider, for example, a mixed-effects logistic-regression model with a per-user random intercept.

If this is overkill, the first of your options makes sense. The second is not so good because impressions aren't independent observations.

  • $\begingroup$ That makes sense. If I measure click-through probability, can I use Pearson's chi-squared test, or should I prefer something else? $\endgroup$
    – gavinmh
    Commented Dec 12, 2016 at 18:20
  • $\begingroup$ @gavinmh The best significance test for a 2 × 2 contingency table is a matter of continuing debate (not to mention the question of whether significance tests are any good in the first place). $\endgroup$ Commented Dec 12, 2016 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.