I have a client that wants to test 3 pricing options to visitors of a website. All options are for the same product and service (no difference except pricing). Their current idea is to present all 3 options to visitors and use conversions to determine which is the right pricing model for their business.

I'm more familiar with split testing, and even then I don't have any experience designing tests outside of interfaces like Optimizely. I'd really like to understand this style of test better, and figure out how to determine the requisite sample size to achieve significant results.

Is there a name for this sort of test? What differences should I be aware of between an A/B test and this scenario?

  • $\begingroup$ Are you saying that a visitor will see one of 3 pricing options based on their business? Or are you saying each visitor will see all three options? In any case, what is your measure? Is it yes-no (purchase or not) or something else? $\endgroup$ – Joel W. Oct 3 '14 at 20:55
  • $\begingroup$ The method you describe has a bit in common with what's sometimes called monadic testing. Sources that go into that might also go into your method. $\endgroup$ – rolando2 Oct 3 '14 at 22:38
  • $\begingroup$ @JoelW. Visitors will see all three options. The measure purchase or not, yes. $\endgroup$ – coreyward Oct 3 '14 at 22:42
  • $\begingroup$ I don't see how you can possibly learn what the profit-maximizing option is from this. Suppose I offer a large pizza for \$10, \$15, or \$20. Everyone picks the lowest price. That tells me nothing about what profits would be if I charged \$20 instead. At best, you get one point on the demand curve at p=$10 and learn nothing about the slope. $\endgroup$ – Dimitriy V. Masterov Oct 5 '14 at 2:34
  • $\begingroup$ @DimitriyV.Masterov That's valid, but we're not looking to determine the optimal price for profit generation — we're testing demand for different pricing structures (e.g. rent vs buy). $\endgroup$ – coreyward Oct 5 '14 at 19:07

A chi-square goodness of fit test seems to be what you are looking for. It will tell you if the null hypothesis of equal preference for each option is tenable. As to sample size, look at this post on power: Power of chi-square test for goodness-of-fit as function of sample size

However, speaking practically, unless the preference for one option is overwhelming, omitting options that are not maximally chosen might still result in lost sales.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.