1
$\begingroup$

We are conducting an A/B test on our hosting site where we sell four different plans (A, B, C, and D). Visitors to our site are randomly assigned to either a base UI (control) or a modified UI (variant). We do not know the visitors' intentions to purchase a specific plan before the assignment.

My question is: After the assignment, is it valid to compare the purchase rates of each plan within each group? For example, can we compare the purchase rates of Plan A between users who saw the original UI and those who saw the modified UI?

Thanks in advance for your insights!

PS: My primary concern is ensuring that any observed changes in purchase behavior are due to the UI change itself and not because the UI change influenced users to switch plans. Specifically, how can we be sure that an increase or decrease in purchases for Plan A, for instance, is due to the UI change and not because the new UI caused users to change their original purchase intentions?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If each visitor purchases only 1 plan, you can make a 4 (plans) by 2 (UI) crosstable, each cell containing the number of purchasers. A common chisquare test could then be used to test if the two UI groups differ significantly in their purchase behaviour.

In case you also have additional, interesting or control, independent variables, a multinomial regression could be used, with the categorical variable "plan" as the dependent variable. This could be useful if you think that the two UI groups differ in terms of important variables that are related to plan preference. Then you typically want to control for such group differences, which is possible with the multinomial regression model.

Such model also allows you to e.g. investigate if the UI influence differs for say men and women, in which case you could model the interaction of UI and gender.

Improve this answer
$\endgroup$
3
  • $\begingroup$ My primary concern is ensuring that any observed changes in purchase behavior are due to the UI change itself and not because the UI change influenced users to switch plans. Specifically, how can we be sure that an increase or decrease in purchases for Plan A, for instance, is due to the UI change and not because the new UI caused users to change their original purchase intentions? $\endgroup$
    – Iman
    Commented Jul 3 at 2:28
  • $\begingroup$ @Iman The added PS in your question is an important one! You say "... increase or decrease ... is due to the UI change and not because the new UI caused users to change their original purchase intentions?". Because you have two DIFFERENT groups of visitors, and each visitor is observed only once. So strictly speaking you do not observe increase or decrease of behaviour over time, but only differences between the two groups of visitors, which can be statistically tested (chisquare test, mulitnomial regression ...) . But the PS suggests a different problem. $\endgroup$
    – BenP
    Commented Jul 3 at 11:26
  • $\begingroup$ Further, you explain what you DO want to test and what NOT. But for me, I'm not an expert in AB experiments, more explanation is needed to better understand the two different things you mention. They sound really similar to me (as a layman), and my wife btw :-) But as you frame it now, I do not think it is possible to extract what you are looking for from the data you have. $\endgroup$
    – BenP
    Commented Jul 3 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.