3
$\begingroup$

I launched a campaign where I give certain users 20% off. This was not launched as A/B test. I’m trying to figure out how to evaluate the incremental impact + ROI of the intervention given that it was not launched as an experiment.

Some solutions I’ve potentially shortlisted are:

  • Synthetic Control - may not be possible because the synthetic control group may too difficult to construct (correct me if I’m wrong on this)
  • ITS - Interrupted Time Series could be tough because it needs at least 8 periods before and 8 weeks after and I can’t wait 8 weeks for my data to come in to evaluate the impact.
$\endgroup$
6
  • $\begingroup$ can you tell us more about how you selected which users got the 20% off? Maybe you already know this and for sure hindsight is 20/20, but if you had just done a little bit of random selection there this would have been straightforward. $\endgroup$ Feb 26 at 0:14
  • 1
    $\begingroup$ Not exactly randomly selected. It’s a group of users from a churn model (XGBoost) that outputs a probability of a user churning. If the probability is above 0.75 we give a coupon. But not everyone redeems the coupon. I wonder if I can make a synthetic control group out of the users that don’t? That would probably be biased since the users who redeem have a higher inclination to make a purchase. $\endgroup$
    – ibarbo
    Feb 26 at 0:21
  • $\begingroup$ You don't mention a target for the intervention, which seems crucial. The lack of randomness in assignment does weaken any conclusions. So, make the most conservative claims about them. You have implicit treatment groups with control=no coupon and treated=coupon. With two groups and two (pre-post) periods, there are four cells. Given that, there are many metrics to estimate the impact of the intervention, e.g., see virtually any text on marketing research. $\endgroup$
    – user78229
    Feb 26 at 1:54
  • $\begingroup$ @MikeHunter apologies. The target is sales. I want to see if giving them the coupon helps users make more purchases in the long run so that we are able to retain them. So the thought would be to have redeemers as treatment and non redeemers as the control and then do a diff in diff to see incrementally i guess. Or do you have some alternate suggestion on the technique I could use? $\endgroup$
    – ibarbo
    Feb 26 at 2:12
  • $\begingroup$ Simpler is always better. I would use one of the many metrics for a design such as yours -- the marketing literature has many examples, I don't have a specific reference with a page number. I've placed one such metric in the Answer section. You can check the calculations yourself. Just plug similar data into a spreadsheet and estimate. $\endgroup$
    – user78229
    Feb 26 at 2:27

2 Answers 2

3
$\begingroup$

If you can't or shouldn't run a randomized trial (i.e., given that your intervention wasn't run as an experiment, as you say), you can consider applying standard techniques for treatment effect estimation in observational data. If I understand you correctly, you're trying to estimate the effect of giving a discount on sales. I'll pretend that we're trying to estimate how much (in dollars) each user would spend, but you can drop in whatever target you want.

There are two commonplace families of methods: weighting and matching. Both require some assumptions, but I'll describe the methods first. Informally, both methods attempt to reweight/resample the treatment and control groups such that they "look like" they were drawn from a randomized controlled trial. I'll try to keep things high-level but feel free to ask for technical details.

Weighting. If you're able to model the probability that you would give each user a discount, you could consider propensity score weighting. The most basic method is IPW/IPTW (inverse probability of treatment weighting):

  1. Choose a set of user characteristics $X$ (e.g., time spent on an app, how many times they've interacted with some page -- you pick these).
  2. Predict the probability that each individual was given a discount (e.g., using logistic regression to predict binary labels, where 1 = discount given and 0 = no discount given). Call this estimate $e(X)$, the propensity score, or the probability of being given a discount given user characteristics $X$.
  3. "Reweight" individuals based on $e(X)$, yielding the estimator

$$\text{Effect of giving discount} = \mathbb{E}\left[\frac{\text{\$ Spent}}{e(X)} \mid \text{Discount given}\right] - \mathbb{E}\left[\frac{\text{\$ Spent}}{1 - e(X)} \mid \text{Discount not given}\right].$$

Looking at the first term on the RHS, we upweight those given a discount with a low probability of actually receiving a discount (and vice versa in those not given a discount).

For example, if $e(X) = 0.1$ (and let's pretend $e(X)$ is well-calibrated), this means that, in expectation, for every individual with characteristics $X$ given a discount, there are nine individuals with those same characteristics $X$ not given a discount. Hence, we upweight those individuals by a factor of 10 ($\frac{1}{0.1}$) in the group given a discount, and by a factor of ~1.1 ($\frac{1}{0.9}$) in the group not given a discount, such that in both groups, we are "mocking-up" a population with the same number of "similar" (in $X$) individuals.

See here for some nice intuition about IPW.

Matching. I'll describe a simple version of matching (nearest-neighbors matching with replacement):

  1. For every individual that you gave a discount, find the individual in the non-discount group that looks the most "similar" (plug in your favorite distance metric for this). That pair of individuals is a match.
  2. Continue until you have matches for everyone in the discount group. You are permitted to pick individuals w/o a discount multiple times (and you can probably guess what happens in the "without replacement" version of matching).

Now, you have two populations that "look similar," and you can estimate the effect of giving a discount in that subpopulation:

$$\text{Effect of giving discount} = \mathbb{E}[\text{\$ Spent} \mid \text{Discount given}] - \mathbb{E}[\text{\$ Spent} \mid \text{Matched group (discount not given)}].$$

However, you do need a few assumptions to hold for these effect estimates to be valid/unbiased:

  1. Consistency. The potential outcome under the observed treatment is the same as the observed outcome.
  2. No unmeasured confounding. Excepting the user characteristics $X$, there are no confounders (think "common causes," or "things that could simultaneously explain changes in likelihood of being given a discount and spending") between treatment and outcome. Also known as "conditional exchangeability/ignorability."
  3. Positivity/overlap. All users have a non-zero chance of being assigned the discount (or not being given the discount).

It's worth justifying to yourself why these hold/don't quite hold for your problem.

Further reading/coding. You should be able to find implementations of each of these in Python and R. Chapter 10 and 11 of the Causal Inference Handbook give some code examples if you're more programming-inclined; otherwise, I'd recommend "What If?" from Hernan and Robins as a reference textbook. If you're curious, more advanced techniques for estimating the effect also exist (e.g., "doubly robust" methods, or see the EconML, causalml, or doubleml Python packages), but those might be overkill for now.

$\endgroup$
1
$\begingroup$

This 'answer' provides the data and calculations discussed in my comments above.

Here's the data, aggregated to the group and period. enter image description here

Here's the resulting metric calculation:

enter image description here

The Post period pooled variance in Excel is calculated as:

(1/POWER(52.71,2))*(POWER(172.19,2)+(POWER(837786,2)*POWER(178.01,2)))

A similar calculation is used for the Pre period pooled variance.

The Pooled Z-test is calculated in Excel as:

-0.21/(POWER((25.05/78758)+(27.00/837786),0.05)

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.