Confidence interval for differences in total sums

I have data from two variations of an ecommerce web site and would like to determine the confidence interval for the difference in generated profit between the variations. My data contains both the order values and the costs (generating support issues) per visitor. 75% of the visitors have been assigned to variation A and the rest to variation B.

I calculate the value of each variation as the sum of order values minus the costs. I would like to find out if there is a difference in the expected value of the variations and a confidence interval for that difference. Of course, the results need to be extrapolated to be comparable, since one variation has fewer visitors.

The value of a variation can be affected both by a change in average order value, change in number of buyers, change in number of support issues resulting in costs, or the average cost of those issues, so there are many parameters to take into account.

My first intention was to calculate a value per visitor, test for mean difference in value per visitor and then extrapolate from there, but I’m not sure that is a valid method. Also, I’m unsure of what test to use as the distribution is then dominated by a huge number of visitors with 0 value (no cost and no order), and costs are typically much smaller than order values which makes the distribution very skewed. Maybe I need to perform some kind of bootstrapping?

The sample size is large, with around 2000 buyers and 100 000 visitors in total. I'm performing the analysis in R, but any help with where to look for a solution is appreciated.

• Couldn't you simply bootstrap this? – Roland Oct 28 '14 at 10:38
• Quite possibly, but in that case, I'm not sure how to go about it. – Mr P Oct 28 '14 at 12:43
• Are average costs and average order value equal in both variations ? – user83346 Oct 30 '15 at 10:52

Yes, you could simply go down the t-test route, because those deviations from normality don't really matter with sample sizes like that. Obviously, bootstrapping is a perfect alternative and I may display how easy that is with the following commented R code:

# examples raw wins in A and be
raw_win_A <- abs(rnorm(100000, mean=5, sd=15))
hist(raw_win_A, xlim=c(-10,100), breaks=20) #skewed
raw_win_B <- abs(rnorm(2000, mean=4.9, sd=20))
hist(raw_win_B, xlim=c(-10,100), breaks=20) #skewed

#compute means of n bootstrap samples of wins in A
n <- 10000
wins_A <- replicate(n, mean(sample(raw_win_A, replace=TRUE)))
#the same with B
wins_B <- replicate(n, mean(sample(raw_win_B, replace=TRUE)))

# show distribution of bootstrapped wins in A and B,
# these aber bound to be normally distributed with increasing n
hist(wins_A)
hist(wins_B)

# show distribution of wins_A minus wins_B
hist(wins_A - wins_B)

cat("Mean of wins_A minus wins-B: ")
cat(mean(wins_A - wins_B))
cat("1.96 times standard deviation of that:")
cat(1.96*sd(wins_A - wins_B))
cat("Confidence interval lower bound: ")
cat(mean(wins_A-wins_B)-1.96*sd(wins_A - wins_B))
cat("Confidence intercal upper bound:")
cat(mean(wins_A-wins_B)+1.96*sd(wins_A - wins_B))
cat("---\n Compare to t-test results:")
print(t.test(raw_win_A, raw_win_B))


This takes some seconds (less than a minute) to run. With the example data simulated in the first few lines I get a bootstrapped conficende/credible interval from -3.973402 to -2.906095 and the t.test function gives a confidence interval from -3.971132 to -2.895014 even though the data are highly skewed (see all those histograms produced by my code). So yes, the t-test is robust against violations from normality whenever n is high enough. The Central Limit Theorem holds.

A straightforward bootstrap approach would be to build a vector of samples (A -B) if A and B are your logs of sales of the different versions of the website. Given this vector you perform bootstrap resamples of this vector. You thus end up with a number of resamples that can approximate the true distribution of the difference. Finally to compute the confidence interval you just use the basic gaussian approximation (or even the empirical percentiles e.g. 5%/95%) of the mean of the bootstrap resamples.