Measuring Advertisement Effect on Sales

Question

I'm currently working on a project focusing on "Measuring the effect of an advertisement on sales of a product." I am seeking advice as I encountered an intricate situation that requires creative solutions.

The situation is as follows:

1. The advertisement campaign was launched across all population (geography, race, gender etc.), making it difficult to establish a straightforward TEST-CONTROL setup for our analysis.

2. Our initial thought process was to forecast post-campaign sales using only pre-campaign data. We considered this forecasted sales data as the 'Counterfactual' sales, representing the sales that would have occurred in the absence of the campaign. Consequently, we treated the forecasted sales as the CONTROL, and the actual post-campaign sales as the TEST.

3. However, we faced a challenge as the actual sales post-campaign declined due to unforeseen external factors (e.g., COVID, financial issues) that were absent before the campaign. This unpredicted decline was not captured by our forecasting model, resulting in projected counterfactual sales consistently exceeding the actual post-campaign sales.

Given this complex scenario, I am eager to hear your valuable suggestions on how we can design an effective observational data study to accurately measure the intended effect of the advertisement on sales. Any insights, alternative methodologies, or creative approaches to tackle this challenge would be greatly appreciated.

Answer 1

You are correct that the ubiquitous launch of this campaign renders a post-hoc A/B test design impossible.

However, pre-post performance test metrics remain an option.

A forecast based on pre-campaign data would define, in marketing terms, a baseline, not a counterfactual, to post-launch sales, assuming ceteris paribus. As you note, all things are not equal in the post period.

Classic marketing mix, ad effectiveness models are rooted in the traditional 4 Ps such as product, price, promotion, place and do not deal well, if at all, with non-marketing vicissitudes such as financial factors, senior debt bond rating changes, brand strategy shifts, manufacturing process changes, much less forces majeures such as covid, lockdowns, etc.

But, there are no rules which say that you can't introduce such metrics as explanatory, exploratory features in your model. As available, widen your metrics to include things like stock market price, dummy variables for covid timelines and anything else which is measurable and represents a potential shift impacting post period performance.