# a/b testing revenue increases

I have 4 geo specific groups. In each of the first three groups, we spent different levels of money in a marketing channel and curtailed all spend in the last. The groups have been designed to be identical in most respects. My goal is compare the lift in revenue (over the forecasted revenue) as a result of spending. What statistical test will work for this comparison?

I am thinking that a non-parametric test such as U test or KS test can be used to compare deltas of revenues (actual - forecasted) between a group with spend and the group with no spend.

Is that thinking appropriate?

• Why do you think that a non-parametric test would be preferable? You may be right, I'm just interested in your line of thinking. Jul 12, 2016 at 12:49
• My rationale is revenues don't follow a normal distribution, so, the deltas won't either. I can make them normal, by bootstrapping 1000 times or by applying a boxcox transformation, but, applying a non-parametric test would be a better option. Jul 13, 2016 at 13:15

The most interpretable, widely used, and easy to implement approach is a simple difference-in-differences model. Here, use linear regression with fixed effects for site (or "geospecific groups" as you call them), fixed effects for continuous amount of spending, and baseline revenue. Model as an outcome the new revenue. Consider log transforming the outcome, since markets, volumes, and other denominators may vary between sites in a way that fixed effects do not explain. Calculate confidence intervals for the "spending" variable to see by what percentage revenue increased for each \$1,000 spent (or other suitable contrast) from baseline.

As an exploratory analysis, calculate interactions between the spending variable and site to see if the trend (if any) is driven by site-specific historical trends. This will assess the validity of using each site as its own historical control, among other possibilities. A limitation is having only 1 control site with 0 spending increase (which is what I assume you mean by curtailment).

Your rationale for using a non-parametric test doesn't make much sense: the comparisons aren't being made explicit, you don't estimate a meaningful effect, and there is a loss of efficiency in using rank based tests when linear models would be even approximately well suited.

In response to my comment you've mentioned that you don't expect the deltas to be normally distributed, because neither of the variables involved in producing delta are normally distributed. That's a fair assumption. I would begin first by reminding you that the assumption behind many parametric statistics is that it is the residuals that are normally distributed, not the variables themselves. You may want to ensure that this is the case before moving on to use non-parametric statistics...

...Now, assuming that you've checked the residuals and they're not normal then your first step would be to use a Kruskal-Wallis test. This is the non-parametric equivalent of a one-way ANOVA, and as an omnibus test this will tell you whether a difference exists somewhere in your groups.

If this is the case then you may want to determine where exactly the difference lies. In order to do this you will want to use a Mann-Whitney U (with appropriate corrections to your alpha level) or, ideally, a Dunn's test to see which groups are different from each other.

• Thanks. At the beginning of the test, we tried to balance the 4 groups in all respects except for the number of stores in each group. Given the revenue data is at the store level, we have different % of revenue lifts for each of the stores. Assuming that K-S test compares ECDFs, is it okay to have different sample sizes for each of the groups? Jul 14, 2016 at 20:09
• What are ECDFs? And by K-S test do you actually mean Kruskal-Wallis? K-S is typically used to refer to Kolmogorov-Smirnov Jul 14, 2016 at 22:16
• No, I meant Kolmogorov-Smirnov Test. By ECDFs, I mean Empirical CDFs. Jul 15, 2016 at 23:04
• Unbalanced sample sizes are typically not a problem for non-parametric tests. Jul 17, 2016 at 17:41