a/b testing revenue increases

Question

I have 4 geo specific groups. In each of the first three groups, we spent different levels of money in a marketing channel and curtailed all spend in the last. The groups have been designed to be identical in most respects. My goal is compare the lift in revenue (over the forecasted revenue) as a result of spending. What statistical test will work for this comparison?

I am thinking that a non-parametric test such as U test or KS test can be used to compare deltas of revenues (actual - forecasted) between a group with spend and the group with no spend.

Is that thinking appropriate?

Answer 1

The most interpretable, widely used, and easy to implement approach is a simple difference-in-differences model. Here, use linear regression with fixed effects for site (or "geospecific groups" as you call them), fixed effects for continuous amount of spending, and baseline revenue. Model as an outcome the new revenue. Consider log transforming the outcome, since markets, volumes, and other denominators may vary between sites in a way that fixed effects do not explain. Calculate confidence intervals for the "spending" variable to see by what percentage revenue increased for each $1,000 spent (or other suitable contrast) from baseline.

As an exploratory analysis, calculate interactions between the spending variable and site to see if the trend (if any) is driven by site-specific historical trends. This will assess the validity of using each site as its own historical control, among other possibilities. A limitation is having only 1 control site with 0 spending increase (which is what I assume you mean by curtailment).

Your rationale for using a non-parametric test doesn't make much sense: the comparisons aren't being made explicit, you don't estimate a meaningful effect, and there is a loss of efficiency in using rank based tests when linear models would be even approximately well suited.

Answer 2

In response to my comment you've mentioned that you don't expect the deltas to be normally distributed, because neither of the variables involved in producing delta are normally distributed. That's a fair assumption. I would begin first by reminding you that the assumption behind many parametric statistics is that it is the residuals that are normally distributed, not the variables themselves. You may want to ensure that this is the case before moving on to use non-parametric statistics...

...Now, assuming that you've checked the residuals and they're not normal then your first step would be to use a Kruskal-Wallis test. This is the non-parametric equivalent of a one-way ANOVA, and as an omnibus test this will tell you whether a difference exists somewhere in your groups.

If this is the case then you may want to determine where exactly the difference lies. In order to do this you will want to use a Mann-Whitney U (with appropriate corrections to your alpha level) or, ideally, a Dunn's test to see which groups are different from each other.