Validity assessment of multiple regression marketing mix model My employer has engaged a consultancy firm to carry out marketing mix modeling in order to quantify the impact of various marketing activities and promotional campaigns on overall sales and also for future marketing budget optimization. As the developed model is considered the analysis firms core asset, we do not have access to it as such, although certain details are shared with us. Given the partial black box nature of the model and my own inexperience, I find it difficult to assess the validity of their model and its predictions. I am therefore reaching out to the Cross Validated community in hope of finding skilled marketing mix modelers who can give some guidance.
The dataset for the analysis consists of weekly data for a period of three years, so in total 156 data points. The model is of multiple linear regression type with target variable chosen to be the total transaction count. Since I haven’t seen the model, I don’t know exactly what regressors are being used and how. At the very least, I believe they should include the following:


*

*Marketing budget per channel (TV, print, out-of-home ads, online video, online display, paid social media, paid search)

*Promotional activities (categorized in 3 dimensions with a total of 5x7x3 combinations)

*Weather

*Marketing budgets for competitors


The model also contains variables to account for monthly seasonality (at least one dummy to control for monthly payday effect), yearly seasonality (probably modeled by a few harmonics, say 3 at the very least) and an overall sales trend. On top of this, something like 5 – 10 holidays must be dummied out. There is also a transformation applied to each of the marketing variables, with two parameters (for each marketing variable) that control marketing response saturation and lagged marketing effects, so called adstock.
To be fair, I want to stress again that I don't know how many of the regressors are actually being used in the model and in what way, but in my mind the large number of (presumed) regressors relative to the dataset length points in the direction of weak statistical support for the regression coefficients or, alternatively, an overfit model whose marketing attribution is in the risk zone of not being trustworthy. I have been shown time series plots of overlaid actual and predicted sales, which suggest that the model fit is extremely good. On the other hand, I am well aware that marketing analysts regularly perform and rely on analyses of this type for budget allocation, which of course suggests a profitable analysis ROI.
Any comments from people with experience in marketing mix modeling and/or general statistics experience are most welcome, in particular concerning the validity of the model estimates in the described setting. Also, suggestions for statistical validity assessment that I can pass on to the analysis firm would be highly appreciated. Here, I am thinking of some sort of test or validity check that can be conducted and presented without the analysis firm handing out the model specifics.
 A: In this case the modellers should be using a time series model and can include mixed-effects to sweep out some of the variation they are not interested in. If they are not using time series, the modelling is a wasted effort as the observations (weeks) are not independent of one another, and therefore violate the underlying tenant of independence. 
To evaluate the predictive capacity of the model we typically use either cross-validation or a boot-strap approach. Cross-validation is a little more common. In the case of time-series, we typically use a rolling window. I.e. fit the model to x time points and then use this model to predict x + 1 - keep doing this till you reach the last data point and then you combine the results. A common and useful metric for this is root mean square error as it is on the original scale, and thus easily interpretable. This directly quantifies the predictive capacity of the model and tells you how well it will extrapolate. You can increase the width of the window (i.e. how well it predicts x + 1, x + 2, x + 3 etc.) to see just how far you can trust the predictions.
Let me know if anything is unclear.
A: I think you can withhold the outcome for the last month, but provide them with the marketing and promotional covariates for those four weeks. Ask them to predict the missing outcomes. Compare their predictions with the actuals that you did not make available. 
This will be a particularly interesting exercise if that last month contains some unusual marketing/promotional behavior relative to what has been done historically. Sometime it is also possible to use "natural" experiments. For example, say your SEM bidding algorithm was broken for a few days or you were off TV, sending spend/impressions to zero. Did their model predict the drop in sales correctly?
A: Cross validation is very hard with time series models. Yes you can hold out the last period but that might be the most important period for your model (recent marketing). Remember MMMs are not forecasting tools but "attribution" models. A simple ARIMA model might produce better forecasts.
Ground truth in MMM is hard. You can simulate some data where you know the causal paths and then let the model estimate the effects, then check against true attribution.
For an open souced model see https://github.com/google/lightweight_mmm
