I have data from two variations of an ecommerce web site and would like to determine the confidence interval for the difference in generated profit between the variations. My data contains both the order values and the costs (generating support issues) per visitor. 75% of the visitors have been assigned to variation A and the rest to variation B.
I calculate the value of each variation as the sum of order values minus the costs. I would like to find out if there is a difference in the expected value of the variations and a confidence interval for that difference. Of course, the results need to be extrapolated to be comparable, since one variation has fewer visitors.
The value of a variation can be affected both by a change in average order value, change in number of buyers, change in number of support issues resulting in costs, or the average cost of those issues, so there are many parameters to take into account.
My first intention was to calculate a value per visitor, test for mean difference in value per visitor and then extrapolate from there, but I’m not sure that is a valid method. Also, I’m unsure of what test to use as the distribution is then dominated by a huge number of visitors with 0 value (no cost and no order), and costs are typically much smaller than order values which makes the distribution very skewed. Maybe I need to perform some kind of bootstrapping?
The sample size is large, with around 2000 buyers and 100 000 visitors in total. I'm performing the analysis in R, but any help with where to look for a solution is appreciated.