0
$\begingroup$

I have results for a a 5 variant pricing test:

A 0% (control group)

B +5%

C -5%

D +10%

E -10%

Below are my results (all normalised to give if exactly 20% of traffic goes through the variant, this is the number of bookings we'd get) variant Bookings

A 100

B 90

C 150

D 60

E 300

If I want to calculate the change in bookings for each % change, how would I do that?

1) +5% change = (B-A)=(90-100)=-10 bookings

or 2) +5% change = (B-A)+(A-C) = (90-100)+(100-150)=-60 (for 2, I have a feeling I should divide the result by 2 (if this is the right way) since otherwise you're double counting the effect of the +5% change) - this method takes into consideration the +5% change from both direction

Or if theres any other statistically correct/better way to calculate this, please let me know.

$\endgroup$
1
$\begingroup$

The relationship between the number of bookings and the price is non-linear in your case. This means that increasing the price by 1% will have a different effect on the price depending on the price you start from. Starting from the price in the control group, an increase in the price seem to have less of an impact than a reduction in the price.

Given the non-linearity, it is difficult to know for sure the effect of a 1% change of the price upwards or downwards. If we do a linear interpolation to estimate the effect of going from A to B and from A to C, it might be better to use the information symmetrically, both from A-B and from A-C. So you could proxy the impact of 1% change of the price as (B-C)/10 = -6. I.e. an increase of the price of 1% around the price in the control group would be associated with a loss of about 6 bookings.

Now, if you wanted to do something more sophisticated, you could also try to fit a polynomial for the relationship between the price. In your case, it looks roughly like the opposite of a cubic polynomial (especially with the number of bookings in logs). If you're ready to make this functional-form assumption, you can refine the estimate of the local average effect of a 1% change.

$\endgroup$
  • $\begingroup$ I'm assuming you're /10 because you're using the +5% and -5% which has a difference of 10? $\endgroup$ – Silver Sep 21 '18 at 22:46
  • $\begingroup$ but wouldnt it be better to use the 0% as baseline case : so we just have +5% change in price gives -10 in bookings? if we use your simple way, then technically +5% means -30 bookings and -5% gives +30 bookings. $\endgroup$ – Silver Sep 21 '18 at 22:48
  • $\begingroup$ Yes, the "/10" is there to scale the coefficients, as you want the effect of a 1% change. Using only B-A to compute the effect of the change at baseline is possible. In general, it is reasonable to assume that the actual slope of the relationship between the price and the number of bookings is going to be something between the slope on the right (-2 = -10/5) and the slope on the left (-5= 50/-5). $\endgroup$ – Roland Sep 22 '18 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.