2
$\begingroup$

This question is a generalization of another question that I asked here.

Suppose that Walmart has 1,000 stores. It has a 20% coupon for cereal, and it hypothesizes that the coupon will increase the sales of cereal by 3%.

Walmart put the coupon in 100 stores on 2022-05-01; the other 900 stores continue to have no coupon. Unfortunately, it does NOT have any sales data from before 2022-05-01. The only data that it has is in the post-intervention period (from 2022-05-01 till today).

Assume that I have data on all the confounding variables that you care about - but ONLY in the post-intervention period.

Given this limitation, is there any method that can estimate the impact of the intervention?

Improve this question
$\endgroup$
5
  • $\begingroup$ Are these 100 stores randomly selected? $\endgroup$
    – Demetri Pananos
    Commented May 8, 2023 at 5:09
  • $\begingroup$ Yes they are. Assume "nice" numbers and "good" conditions for everything. I'm trying to pinpoint the problem to JUST the lack of pre-intervention data. $\endgroup$
    – Iterator516
    Commented May 8, 2023 at 13:36
  • $\begingroup$ So I understand you just don't have outcome (sales) at baseline. That should be ok for a randomized controlled experiment. Might be less statistically efficient (including outcome pre-treatment can account for lots of the variance in outcome post-treatment and therefore make the treatment effect estimation more precise), but that's not necessary for unbiased effect estimation. $\endgroup$
    – ehudk
    Commented May 11, 2023 at 7:10
  • $\begingroup$ I have edited my question to specify that the data for confounders are available in ONLY the post-intervention period. Does that change your comment, @ehudk? $\endgroup$
    – Iterator516
    Commented May 11, 2023 at 14:09
  • 1
    $\begingroup$ @Iterator516, unfortunately yes. Confounders should cause the treatment and outcome and therefore they must precede both. You will need unreasonable assumptions to claim post-treatment variables can be valid confounders. Moreover, adjusting for post-treatment factors usually only creates more bias since you might adjust on a mediator or a collider. $\endgroup$
    – ehudk
    Commented May 14, 2023 at 6:53

1 Answer 1

Reset to default
1
$\begingroup$

If treated stores are (truly) randomly selected, this is an experimental setup, and you can derive ATT straightforwardly in a number of ways. Random selection with an appropriate sample size does a better job of eliminating the heterogeneity than other methods that use pre-treatment data to build a synthetic counterfactual to derive the ATT. Additional variables related to confounding effects might help reveal nuanced relationships and help ensure that the relationships are not spurious (a sample of 100 of 1,000 stores is reasonably safe in this regard, IMO), but the randomization is doing the heavy lifting. You would not need pre-treatment sales data to get a reasonable estimate of the ATT.

With knowledge (that I don't have) about the expected duration of the coupon's effect on sales; you can pick an appropriate post-treatment time period/cutoff (or several like short-, medium-, long-term); aggregate the sales or number sold per-store for the item among each of the treatment group and control group members, respectively, during the period; and use an appropriate model to get the difference in means. If the distributions of sales are appropriate/normal, a simple t-test or regression model would do the trick. Otherwise, you could take the natural log of the sales or use an appropriate GLM, like Poisson or negative binomial.

Improve this answer
$\endgroup$
3
  • $\begingroup$ For the sake of simplicity, let's assume non-random sampling. I want to pinpoint the problem to JUST the lack of pre-intervention data, and I want to use causal inference. (Furthermore, I don't think that random sampling can help in the case of retail stores; they are just WAAAYYYYY too heterogeneous: climate, driving distance, walking distance, store manager, neighborhood income, store size, store design, shopper demographics, just to name A FEW.) $\endgroup$
    – Iterator516
    Commented May 10, 2023 at 23:15
  • 1
    $\begingroup$ I think randomization would work well if your sample is similar to 1,000. It's the gold standard for getting ATT on people, and they are quite heterogenous. If they are not randomly sampled, you could do your best to perform matching on treatment and control based on known attributes and get an ATT. The more attributes you have and the more potential control cases you have to find good matches, the better your estimate should be. But randomization would be better, and checking for pre-treatment parallel trends with (without) matching would be second (third) best. What are your options? $\endgroup$
    – dcoy
    Commented May 11, 2023 at 1:10
  • $\begingroup$ A) Unfortunately, I can't check for pre-treatment parallel trends, because I don't have pre-treatment data. B) Randomization is NOT an option. $\endgroup$
    – Iterator516
    Commented May 11, 2023 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.