0
$\begingroup$

My employer has engaged a consultancy firm to carry out marketing mix modeling in order to quantify the impact of various marketing activities and promotional campaigns on overall sales and also for future marketing budget optimization. As the developed model is considered the analysis firms core asset, we do not have access to it as such, although certain details are shared with us. Given the partial black box nature of the model and my own inexperience, I find it difficult to assess the validity of their model and its predictions. I am therefore reaching out to the Cross Validated community in hope of finding skilled marketing mix modelers who can give some guidance.

The dataset for the analysis consists of weekly data for a period of three years, so in total 156 data points. The model is of multiple linear regression type with target variable chosen to be the total transaction count. Since I haven’t seen the model, I don’t know exactly what regressors are being used and how. At the very least, I believe they should include the following:

  • Marketing budget per channel (TV, print, out-of-home ads, online video, online display, paid social media, paid search)
  • Promotional activities (categorized in 3 dimensions with a total of 5x7x3 combinations)
  • Weather
  • Marketing budgets for competitors

The model also contains variables to account for monthly seasonality (at least one dummy to control for monthly payday effect), yearly seasonality (probably modeled by a few harmonics, say 3 at the very least) and an overall sales trend. On top of this, something like 5 – 10 holidays must be dummied out. There is also a transformation applied to each of the marketing variables, with two parameters (for each marketing variable) that control marketing response saturation and lagged marketing effects, so called adstock.

To be fair, I want to stress again that I don't know how many of the regressors are actually being used in the model and in what way, but in my mind the large number of (presumed) regressors relative to the dataset length points in the direction of weak statistical support for the regression coefficients or, alternatively, an overfit model whose marketing attribution is in the risk zone of not being trustworthy. I have been shown time series plots of overlaid actual and predicted sales, which suggest that the model fit is extremely good. On the other hand, I am well aware that marketing analysts regularly perform and rely on analyses of this type for budget allocation, which of course suggests a profitable analysis ROI.

Any comments from people with experience in marketing mix modeling and/or general statistics experience are most welcome, in particular concerning the validity of the model estimates in the described setting. Also, suggestions for statistical validity assessment that I can pass on to the analysis firm would be highly appreciated. Here, I am thinking of some sort of test or validity check that can be conducted and presented without the analysis firm handing out the model specifics.

Improve this question
$\endgroup$
0
$\begingroup$

I think you can withhold the outcome for the last month, but provide them with the marketing and promotional covariates for those four weeks. Ask them to predict the missing outcomes. Compare their predictions with the actuals that you did not make available.

This will be a particularly interesting exercise if that last month contains some unusual marketing/promotional behavior relative to what has been done historically. Sometime it is also possible to use "natural" experiments. For example, say your SEM bidding algorithm was broken for a few days or you were off TV, sending spend/impressions to zero. Did their model predict the drop in sales correctly?

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks! What you suggest is a smart strategy and I will investigate whether or not it is a feasible approach within this project. Your reply does answer my call for a suitable model assessment test, and I will therefore accept it. However, I still hope for further comments on model validity given the description above (in particular with regards to the seemingly few data points per variable). Maybe you have some general ideas about this too? $\endgroup$ – Robert May 10 '19 at 7:12
  • $\begingroup$ That is definitely a lot of parameters given 156 points. The test I proposed will reveal if the model is overfit and the good predictive performance you see is illusory. They may also tell you that they applied some sort of variable selection procedure, but I would still worry about allocation decisions made from a model geared towards prediction. I am a big fan of experimentation for learning about this, but that is slower and expensive. However, there have been several examples in marketing where the experiment gives different answers from the observational methods. $\endgroup$ – Dimitriy V. Masterov May 10 '19 at 15:41
  • $\begingroup$ That is my impression too (that there are alarmingly few data points per parameter). Way below F Harrell’s rule of thumb in his book on regression, 10-20. So I’m wondering if I’ve missed something. If perhaps the two parameters for saturation/adstock shouldn’t count as two, if seasonality dummies should count as fewer than their actual number etc. That was my main reason for reaching out to the marketing model community. Although I really can’t see any rationale for such a view. $\endgroup$ – Robert May 12 '19 at 7:05
  • $\begingroup$ Regarding your comment on the model being geared towards prediction, a recurring argument from the analyst firm seems to be that overfitting is not so much an issue because we are mainly using the model for inference on historical data. Maybe you can help me understand that? To my understanding, overfitting would deteriorate the model inference too, not just its predictability. $\endgroup$ – Robert May 12 '19 at 7:21
  • $\begingroup$ Interesting thoughts on experiments vs observation. I’d like to find out more! Do you have any resources (articles, blogs etc) to share with examples? $\endgroup$ – Robert May 13 '19 at 6:24
0
$\begingroup$

In this case the modellers should be using a time series model and can include mixed-effects to sweep out some of the variation they are not interested in. If they are not using time series, the modelling is a wasted effort as the observations (weeks) are not independent of one another, and therefore violate the underlying tenant of independence.

To evaluate the predictive capacity of the model we typically use either cross-validation or a boot-strap approach. Cross-validation is a little more common. In the case of time-series, we typically use a rolling window. I.e. fit the model to x time points and then use this model to predict x + 1 - keep doing this till you reach the last data point and then you combine the results. A common and useful metric for this is root mean square error as it is on the original scale, and thus easily interpretable. This directly quantifies the predictive capacity of the model and tells you how well it will extrapolate. You can increase the width of the window (i.e. how well it predicts x + 1, x + 2, x + 3 etc.) to see just how far you can trust the predictions.

Let me know if anything is unclear.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your reply! I believe time series methods are in use, but neither cross validation nor bootstrapping for model assessment. Do you have any general ideas on how well supported the inference can possibly be given the (sparse) model description above, or any suggestions for tests that can be conducted on the finalized model? (I have accepted Dimitriy's answer below, which suggests using newer hold-out data) $\endgroup$ – Robert May 10 '19 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.