How to evaluate AB test for determining price elasticity?

Question

Given a standard AB test setup on an e-commerce platform with users randomly seeing same product priced either with control price or a variant, how does one correctly assess the results?

Say, after a week we have X sales for control and Y sales for variant. We are not looking at conversion rates, all we are interested in is the sales uplift from one discount point to another. How do we know if the difference that we observe is statistically significant (Chi squared goodness of fit?) and that the total sales are enough to trust the observed uplift with a given confidence level?

Answer 1

Bootstrap is a generic and simple method for your question.

assume your raw data like this:

• randomization unit is user_id: each user have a consistent treatment and may have multiple visits/transactions, ending with a purchase or not

• y can be purchase amount or is purchased(0/1)

library(boot)
library(tidyverse)

df_raw <- tibble(
  user_id = c(1, 1, 1, 2, 3, 3, 4, 5, 5, 6),
  treat   = c(0, 0, 0, 1, 0, 0, 0, 1, 1, 0),
  y       = c(0, 100, 0, 30, 0, 0, 0, 10, 0, 0)
)

# experiment users
df_user <- distinct(df_raw, user_id)

    # A tibble: 10 × 3
       user_id treat     y
         <dbl> <dbl> <dbl>
     1       1     0     0
     2       1     0   100
     3       1     0     0
     4       2     1    30
     5       3     0     0
     6       3     0     0
     7       4     0     0
     8       5     1    10
     9       5     1     0
    10       6     0     0

# This is a little different from the standard bootstrap t-test, which fixes the A/B group sample size, but more like a case resample bootstrap regression
get_stat <- function(df_user, inds, df_raw) {
  df_raw %>%
    # we resample user not transaction, for the randomization unit is user
    inner_join(df_user[inds, ], by = 'user_id') %>%
    summarise(
      # transaction level
      transaction_cnt_0 = sum(if_else(treat == 0, 1, 0)),
      transaction_cnt_1 = sum(if_else(treat == 1, 1, 0)),

      # user level
      user_cnt_0 = n_distinct(if_else(treat == 0, user_id, NA_real_)),
      user_cnt_1 = n_distinct(if_else(treat == 1, user_id, NA_real_)),

      y0 = sum(if_else(treat == 0, y, 0)),
      y1 = sum(if_else(treat == 1, y, 0)),

      # y0 = max(if_else(treat == 0, y, 0)),
      # y1 = max(if_else(treat == 1, y, 0)),

      # average sales per user
      diff_user = y1 / user_cnt_1 - y0 / user_cnt_0,

      # average sales per transaction
      diff_transaction = y1 / transaction_cnt_1 - y0 / transaction_cnt_0
    ) %>%

    pull(diff_user)

}

boot_res <- boot(df_user, get_stat, R = 1000, df_raw = df_raw)
boot_res

ORDINARY NONPARAMETRIC BOOTSTRAP


Call:
boot(data = df_user, statistic = get_stat, R = 1000, df_raw = df_raw)


Bootstrap Statistics :
    original  bias    std. error
t1* -6.66667  -4.785     32.3452

# if you hava a large sample, this may hava compute problems
# boot.ci(boot_res, type = 'bca')

boot.ci(boot_res, type = 'basic')
```