Requests for a quote come in completely independently, and our system assigns each to either A or B randomly (50/50% split). B has slightly different logic, that is what we are testing.

Quotes usually aren't purchased (let's call it a \$0 purchase), but when they are it's usually for < $2 (this happens around 0.01% of the time).

I would like to be able to calculate for our experiment how much Group B's new logic changed (if at all) the revenue (sum of the purchase prices).

I am not a stats expert (please, ELI5!), and so I'm not sure about how to set up the hypothesis testing / calculations.

Purchase price is not normally distributed, so perhaps a non-parametric Mann-Whitney is best, but scipy really only gives a way to calculate the p-value. I care about the p-value, but I'm really more curious about the effect difference.

Here's a histogram of the purchase prices (with no purchase as \$0):

enter image description here

Practically, I'm looking for a straightforward way to do this in Python.

Attempt with a t-test as suggested:

from scipy import stats

df = get_data_from_db(experiment_name='blah')

# two columns: price, name of group
a = df[(df.price > 0) & (df.name == 'A')].price.values
b = df[(df.price > 0) & (df.name == 'B')].price.values

# probability of sale (non-zeros only!)
p_a = len(a) / len(df)
p_b = len(b) / len(df)

statistic, pvalue = stats.ttest_ind(a, b, equal_var=False)

print(f'N={len(df)}, N_a={len(a)}, N_b={len(b)}, p_a={round(p_a, 5)}, p_b={round(p_b, 5)}')
print(f'statistic={round(statistic, 2)}, pvalue={round(pvalue, 5)}')


N=1894166, N_a=116, N_b=178, p_a=6e-05, p_b=9e-05
statistic=-3.09, pvalue=0.0022
  • $\begingroup$ Can you share the data? $\endgroup$ Dec 9, 2019 at 1:26
  • $\begingroup$ sure, what format would be best? it's quite sparse, though. 1.8M samples with about 300 non-zero prices. or perhaps you just meant some statistics on the samples? $\endgroup$ Dec 9, 2019 at 1:29
  • $\begingroup$ Oh wow that is sparse. Maybe I'll just cook up some data to demonstrate the approach. $\endgroup$ Dec 9, 2019 at 1:30
  • $\begingroup$ Yeah the histogram I posted is actually not the same data, but it's close enough to show the shape. Sorry to confuse! $\endgroup$ Dec 9, 2019 at 1:31
  • $\begingroup$ I could post the non-zero ones, and the counts of zeros for each group if that might make it a bit more concrete. $\endgroup$ Dec 9, 2019 at 1:32

1 Answer 1


I wasn't happy with my old answer. Here is a better one in my opinion.

I would like to be able to calculate for our experiment how much Group B's new logic changed (if at all) the revenue (sum of the purchase prices).

Ignoring the \$0 purchases, let the $i^{th}$ customer's purchase in group $X$ be denoted by $y_{x,i}$. The revenue from group $X$ is then $\sum_i y_{x,i} = R_x$. If you want to know how much group B's logic changed the revenue, then you should examine the expected revenue per 100,000 purchasing customers (or some other number).

We have theorems about how the revenue (sums of iid random variables) are distributed. In short, the revenue per N customers is

$$R_x \sim \mathcal{N}(N\mu_x, N\sigma^2_x)$$

Where $\mu_x$ and $\sigma^2_x$ are the mean and variance of $y_{x,i}$.

The distribution of the differences in revenue is then

$$R_B - R_A \sim \mathcal{N}(N\mu_B - N\mu_a, N\sigma^2_B + N \sigma^2_A)$$

From here, you can compute a 95% confidence interval for the difference in revenue. You don't have access to the $\mu$'s and $\sigma$'s, but you can estimate them from your data. Here is some python code simulating and verifying these results

import numpy as np
from scipy.stats import gamma, norm
import matplotlib.pyplot as plt

N = 1000 #1000 customers
n_sims= 1000 #Number of simulations

# Difference in means of 1 unit

groupA = gamma(10)
A = groupA.rvs(size = (n_sims, N))

groupB = gamma(11)
B = groupB.rvs(size = (n_sims, N))

# Distrbution of differences in revenue
D = B.sum(axis=1) - A.sum(axis=1)

# Now, plot on top the normal distribution
fig, ax = plt.subplots(dpi = 120)
plt.hist(D, edgecolor = 'white', bins = 20, density = True)

x = np.linspace(600,1400)
true_dens = norm(loc = N*groupB.mean() - N*groupA.mean(), scale = np.sqrt(N*groupA.var() + N*groupB.var()))
est_dens = norm(loc = D.mean(), scale = D.std())
ax.plot(x, true_dens.pdf(x), color = 'red', label = 'True Dist')
ax.plot(x, est_dens.pdf(x), color = 'C1', label = 'Estimated Dist')
plt.title('Difference in Revenue')
plt.xlabel('Revenue from B - Revenue from A')

enter image description here

In this example, I have simulated your experiment 1000 times. Customer purchase amounts are drawn from a gamma distrubition, and the mean of this distribution changes with the group. Each time, the difference in revenue is computed and plotted as a histogram.

In this simulation, we would expect group B to yield an additional $1000 in revenue over group A per 1000 customers. That makes sense; the real difference between groups is 1 unit. However, because of the uncertainty in the data, it would be best to instead give a confidence interval rather than a single number. Luckily, this can be done by computing...

est_dens.ppf([0.025, 0.975])

I still get the feeling this isn't exactly what you want, so just let me know if this helps at all.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.