# Calculate statistical significance in A/B price test

Let's assume I'm selling a product at two price points, A and B and I'm alternating the displayed price to each visitor for a period of time. I receive a specific number of orders, O(a) and O(b) at each price point after time T.

If I compare the profit after time T of the orders at each price point, is there a way to determine statistical significance?

I do not have conversion rates in this case. I simply know the price points, time elapsed, and number of orders at each price point.

For example, users order 15 widgets at a 23 dollar price point and 17 widgets at a 24 dollar price point after 3 days. Is this enough information to run analysis?

Edit:

For simplicity, assume each selling price is also the profit (cost of goods sold is 0).

• I would say it's obviously not sufficient, since the profit cannot be determined from just this information. – whuber Sep 15 '15 at 18:08
• I have the profit and cost of goods sold. Let's assume the price point is the profit in this case for simplicity. – jheathco Sep 15 '15 at 18:11
• @whuber Another thought (read, "question") is, what about using a Poisson generalized linear model... Not to compare the numbers 15 to 17, as initially stated... – Antoni Parellada Sep 15 '15 at 23:31
• @Antoni Could you explain what a c-test is? (It doesn't appear to refer to Cochran's C-test.) – whuber Sep 15 '15 at 23:46
• @whuber Just learned of its existence today - I landed on this post with an answer by Glen. I downloaded the paper, and spent quite some time trying to make sense of it. In the end Glen makes it very simple - just a binomial with a ratio of the rate parameters as the conditional probability... Something like this... – Antoni Parellada Sep 15 '15 at 23:50

This is definitely not sufficient. In order to test the effectiveness of the price change you made, you need the total number of visitors visiting the site during those times who would see each of the prices.

Using that information you could instead do an A/B test comparing the percentage of users who purchase widgets (a much more informative metric that would account for increased traffic to the site).

Once you determine that you can measure the significance using the following formula: As with most analysis questions, the answer is you can definitely test something, but whether or not the test will really address your underlying research question is debatable. It will rely on assumptions that may not be met in this case, and it sounds like you don't have the information to be able to either run a more careful test or to investigate the assumptions directly. That said, here's an approach you could take:

Assume that there is some expected number of purchases per time interval T, and that the expected number of purchases per T may be different depending on the price. If your data extend over multiple T intervals, you can test whether there is a difference in expected number of purchases for price A vs. price B with a poisson regression. Your sample size will be the number of T intervals (e.g. if T is one day, then N = the number of days). The outcome variable would be the number of purchases, and the predictor would be price (A or B).

This approach makes the following (potentially problematic) assumptions:

1. That there are no additional variables that would predict number of purchases other than price. For example, this implies that the general volume of traffic either doesn't matter or is the steady over time (i.e. it's the same for each interval T), and it doesn't vary by price.
2. That number of purchases during each interval T are independent of previous purchases. In particular, if the same individual visits more than once, whether or not they buy depends only on the current price they see and not on the history of prices they've been exposed to.

To the extent that these assumptions are violated, the results of the analysis become less relevant/informative.