I would like to run a statistically rigorous experiment, similar to that of an E-commerce A/B test.

I want to create a checkerboard of time periods where I alternate between treatments A and B. Ultimately I care about increasing revenue.

I believe the treatment could affect any or all of the following:

  1. number of visitors per period
  2. percent of visitors who become customers
  3. the revenue per customer
  4. the types of customers and their spending profiles (e.g. big customers, small customers, customers interested in specific products spend different amounts)

Hence I feel that my ideal response is period revenue, i.e. the total revenue over the period of each treatment being in effect (so if I have 2 treatments with 10 periods each, I have 20 responses).

But since period revenue is really a sum over individual order revenues (which have large variance) and is affected by visitor frequency and customer/visitor ratio etc., I know that the period revenue will be quite noisy (variance of sum >= sum of variances, assume not negatively correlated). Therefore I'm worried about the power of my experiment and whether I will actually be able to find significance.

To address this, I see the following options:

  1. Shorten the periods? If I want the whole experiment to take a fixed amount of time, like 2 weeks, I think this would give me more data points while reducing the variance of the response. But I'm not certain how to determine the ideal period length and determine how short is too short.
  2. Replace period revenue as the response with one of the metrics from the list above with a reasonable variance / historical distribution. Downside e.g.: if we pick # of visitors as the response and find a significant effect, that may not mean revenue increased, maybe customer to visitor ratio decreased.
  3. Run an analysis for each of the metrics above separately. Since running more comparisons and p-values, would need to correct the p-values with Bonferroni or similar. Downside e.g.: if revenue per customer increases but customers go down, it's not clear my revenue actually increased.
  4. Build some sort of composite response variable. I feel like the revenue is already effectively a composite variable. Similarly, I'd expect summing/differencing these variables will result in more variance, not less variance, so I'm not sure my power would increase.

Which of the following approaches seem most reasonable? Am I missing some possible solutions? What additional exploratory work should I do to determine my approach? Are there research papers involving similar types of experiments that I can reference (I'm struggling to find the right keywords to form an effective search)?

Thanks! Let me know if I can add any detail.


1 Answer 1


OK, here is my two cents based on your framing.

Variance is a factor, yes, but so is underdetermination of the effect. You mention this in your second point (e.g. two metrics could change in such a way that they cancel one another out).

To combat this, here is some general advice:

  • Try to explain away as much variance as you can. If your unit of randomization is the user, it makes sense to use features about the user which are related to their spend. One way to do this is to use CUPED/ANCOVA and create a variable for their spend prior to the experiment starting.

  • Your metric should be determined by the unit of randomization. I don't think it makes sense to randomize users and then make revenue per day your metric of interest. Your metric can be partly decided based on what the intervention is designed to do. For example, do I think that a change on the landing page is going to effect behaviour at checkout? Depends on how far down the funnel checkout is.

  • Revenue is a KPI that can be factored in a bunch of ways. For example, revenue could be Revenue per converted user x conversion rate x number of users landing per day x number of days. It is fine to Pick a primary metric in according to point two and then make all of the other metrics secondary metrics. This can give you insight into what is driving the change, but can be hard to interpret in the language of frequentist inference because most people operate on a reject/fail to reject basis (e.g. why did revenue decrease, all my other metrics show no change p>0.05). There is likely no need to adjust the p value here unless you think the number of users converting at one stage effects the conversion rate at the next stage. E.g. Does conversion rate effect revenue per converted user? No, those are likely independent. People do not likely spend more just because more people are choosing to spend (except in some pathological cases like we saw with Covid and toilet paper, I think this probably doesn't apply to e-commerce but you can make that decision for yourself).

  • $\begingroup$ Thanks! I will look at CUPED/ANCOVA, looks very relevant. I am technically not working with E-Commerce but using it as an example. Thus I don't have the ability to randomize users, I think the only thing I can reliably randomize is blocks of time. Does that change your answer? $\endgroup$
    – wahsmail
    Commented Jan 6, 2023 at 18:27
  • $\begingroup$ @wahsmail only slightly. Techniques like "Randomized Block Designs" will be relevant. Your main metric might now be Revenue per day, and the block design will allow you to adjust for "Day of Week" type effects. You might be able to use "Last Week's revenue" as a possible tool for ANCOVA. $\endgroup$ Commented Jan 6, 2023 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.