How to evaluate the impact of an intervention with no control group?

Question

I launched a campaign where I give certain users 20% off. This was not launched as A/B test. I’m trying to figure out how to evaluate the incremental impact + ROI of the intervention given that it was not launched as an experiment.

Some solutions I’ve potentially shortlisted are:

• Synthetic Control - may not be possible because the synthetic control group may too difficult to construct (correct me if I’m wrong on this)
• ITS - Interrupted Time Series could be tough because it needs at least 8 periods before and 8 weeks after and I can’t wait 8 weeks for my data to come in to evaluate the impact.

Answer 1

If you can't or shouldn't run a randomized trial (i.e., given that your intervention wasn't run as an experiment, as you say), you can consider applying standard techniques for treatment effect estimation in observational data. If I understand you correctly, you're trying to estimate the effect of giving a discount on sales. I'll pretend that we're trying to estimate how much (in dollars) each user would spend, but you can drop in whatever target you want.

There are two commonplace families of methods: weighting and matching. Both require some assumptions, but I'll describe the methods first. Informally, both methods attempt to reweight/resample the treatment and control groups such that they "look like" they were drawn from a randomized controlled trial. I'll try to keep things high-level but feel free to ask for technical details.

Weighting. If you're able to model the probability that you would give each user a discount, you could consider propensity score weighting. The most basic method is IPW/IPTW (inverse probability of treatment weighting):

1. Choose a set of user characteristics $X$ (e.g., time spent on an app, how many times they've interacted with some page -- you pick these).
2. Predict the probability that each individual was given a discount (e.g., using logistic regression to predict binary labels, where 1 = discount given and 0 = no discount given). Call this estimate $e(X)$, the propensity score, or the probability of being given a discount given user characteristics $X$.
3. "Reweight" individuals based on $e(X)$, yielding the estimator

$$\text{Effect of giving discount} = \mathbb{E}\left[\frac{\text{\$ Spent}}{e(X)} \mid \text{Discount given}\right] - \mathbb{E}\left[\frac{\text{\$ Spent}}{1 - e(X)} \mid \text{Discount not given}\right].$$

Looking at the first term on the RHS, we upweight those given a discount with a low probability of actually receiving a discount (and vice versa in those not given a discount).

For example, if $e(X) = 0.1$ (and let's pretend $e(X)$ is well-calibrated), this means that, in expectation, for every individual with characteristics $X$ given a discount, there are nine individuals with those same characteristics $X$ not given a discount. Hence, we upweight those individuals by a factor of 10 ($\frac{1}{0.1}$) in the group given a discount, and by a factor of ~1.1 ($\frac{1}{0.9}$) in the group not given a discount, such that in both groups, we are "mocking-up" a population with the same number of "similar" (in $X$) individuals.

See here for some nice intuition about IPW.

Matching. I'll describe a simple version of matching (nearest-neighbors matching with replacement):

1. For every individual that you gave a discount, find the individual in the non-discount group that looks the most "similar" (plug in your favorite distance metric for this). That pair of individuals is a match.
2. Continue until you have matches for everyone in the discount group. You are permitted to pick individuals w/o a discount multiple times (and you can probably guess what happens in the "without replacement" version of matching).

Now, you have two populations that "look similar," and you can estimate the effect of giving a discount in that subpopulation:

$$\text{Effect of giving discount} = \mathbb{E}[\text{\$ Spent} \mid \text{Discount given}] - \mathbb{E}[\text{\$ Spent} \mid \text{Matched group (discount not given)}].$$

However, you do need a few assumptions to hold for these effect estimates to be valid/unbiased:

1. Consistency. The potential outcome under the observed treatment is the same as the observed outcome.
2. No unmeasured confounding. Excepting the user characteristics $X$, there are no confounders (think "common causes," or "things that could simultaneously explain changes in likelihood of being given a discount and spending") between treatment and outcome. Also known as "conditional exchangeability/ignorability."
3. Positivity/overlap. All users have a non-zero chance of being assigned the discount (or not being given the discount).

It's worth justifying to yourself why these hold/don't quite hold for your problem.

Further reading/coding. You should be able to find implementations of each of these in Python and R. Chapter 10 and 11 of the Causal Inference Handbook give some code examples if you're more programming-inclined; otherwise, I'd recommend "What If?" from Hernan and Robins as a reference textbook. If you're curious, more advanced techniques for estimating the effect also exist (e.g., "doubly robust" methods, or see the EconML, causalml, or doubleml Python packages), but those might be overkill for now.

Answer 2

This 'answer' provides the data and calculations discussed in my comments above.

Here's the data, aggregated to the group and period.

Here's the resulting metric calculation:

The Post period pooled variance in Excel is calculated as:

(1/POWER(52.71,2))*(POWER(172.19,2)+(POWER(837786,2)*POWER(178.01,2)))

A similar calculation is used for the Pre period pooled variance.

The Pooled Z-test is calculated in Excel as:

-0.21/(POWER((25.05/78758)+(27.00/837786),0.05)