My employer has engaged a consultancy firm to carry out marketing mix modeling in order to quantify the impact of various marketing activities and promotional campaigns on overall sales and also for future marketing budget optimization. As the developed model is considered the analysis firms core asset, we do not have access to it as such, although certain details are shared with us. Given the partial black box nature of the model and my own inexperience, I find it difficult to assess the validity of their model and its predictions. I am therefore reaching out to the Cross Validated community in hope of finding skilled marketing mix modelers who can give some guidance.
The dataset for the analysis consists of weekly data for a period of three years, so in total 156 data points. The model is of multiple linear regression type with target variable chosen to be the total transaction count. Since I haven’t seen the model, I don’t know exactly what regressors are being used and how. At the very least, I believe they should include the following:
- Marketing budget per channel (TV, print, out-of-home ads, online video, online display, paid social media, paid search)
- Promotional activities (categorized in 3 dimensions with a total of 5x7x3 combinations)
- Weather
- Marketing budgets for competitors
The model also contains variables to account for monthly seasonality (at least one dummy to control for monthly payday effect), yearly seasonality (probably modeled by a few harmonics, say 3 at the very least) and an overall sales trend. On top of this, something like 5 – 10 holidays must be dummied out. There is also a transformation applied to each of the marketing variables, with two parameters (for each marketing variable) that control marketing response saturation and lagged marketing effects, so called adstock.
To be fair, I want to stress again that I don't know how many of the regressors are actually being used in the model and in what way, but in my mind the large number of (presumed) regressors relative to the dataset length points in the direction of weak statistical support for the regression coefficients or, alternatively, an overfit model whose marketing attribution is in the risk zone of not being trustworthy. I have been shown time series plots of overlaid actual and predicted sales, which suggest that the model fit is extremely good. On the other hand, I am well aware that marketing analysts regularly perform and rely on analyses of this type for budget allocation, which of course suggests a profitable analysis ROI.
Any comments from people with experience in marketing mix modeling and/or general statistics experience are most welcome, in particular concerning the validity of the model estimates in the described setting. Also, suggestions for statistical validity assessment that I can pass on to the analysis firm would be highly appreciated. Here, I am thinking of some sort of test or validity check that can be conducted and presented without the analysis firm handing out the model specifics.