I'm relatively new to machine learning. I have come across the "ML is for prediction not inference” statement but this did not really sink in until my current project, a marketing mix modelling problem.
In a marketing context, I am trying to predict successes (orders) that will occur within the next 28 days based on the marketing activity done today. I have experimented with several configurations (different algorithms, different feature sets) of which a good handful give reasonable and fairly similar mean/median accuracy in cross validation.
However, for us as a business, probably even more important than accuracy is getting a reasonable read of each marketing channel's contribution to the outcome. I've approached this using sensitivity analysis where I set all features related to the channel to 0 and measure the difference in predicted outcome i.e
channel contribution = predictions where channel X is present - predictions where channel x is absent
Even though many configurations have fairly similar (definitely not identical) accuracy, the channel contribution varies wildly.
See below predictions and contributions made on truly unseen data.
Notes: purples= boosted trees greens= random forest the rest are a mix of Lasso/ElasticNet/Ridge.
No polynomial features are included. All trained on exactly the same data - a seed ensures same training observations are used in every configuration. The feature sets were chosen carefully using domain knowledge (i.e. I have not thrown the kitchen sink in).
Overall predictions made by a variety of models (different algorithms / feature set combinations). I've included all in this screenshot to show variance. The black line represents the realised actuals. https://www.evernote.com/l/AJI1cQE5O1lEVLAv8a9uW0mxemjW5N3IKms
Contribution of channel X as determined through sensitivity analysis. You can see how wildly it can vary at times between configurations. https://www.evernote.com/l/AJLw2jRjZ6lNCrxRUkJZvJ_JZFzxQ6nokoY
As there is no ground truth for the contribution of each channel to calibrate against, are there any guidelines/best practices that help pick a model when the goal is both reasonable overall prediction AND realistic read on contribution of each channel? I realise that this may be a big ask and I go back to "prediction not inference" mantra but in order to support initiatives and decisions in the real business world (not "computer says no" types of decisions but "change X to influence outcome Y”), I see no other way for ML to add value but to reasonably quantify the influence of X. And if different models have such different views of the problem and how each feature contributes to it, it's very unclear to me how to use ML for these kinds of real world business problems.
As a side note, I don't see this as purely a ML problem as this could reasonably happen with Linear Regression too unless I am mistaken (3 out of the 5 algorithms used are generalised linear models).
I'm currently leaning between:
picking the model with the lowest cross validation score on overall prediction. Such a hard rule breaks down when the resulting channel contribution is so wildly different to what business stakeholders see as being reasonable and what other sources of information indicate (in this case, multi touch attribution).
picking the model that gives a low cross validation score (but perhaps not lowest) but uses the feature set that is seen by business stakeholders with expert domain knowledge as the most reasonable one (so the choice of model is used without seeing channel contribution upfront).
I would appreciate some thoughts on:
Is my approach misguided somehow? If yes, then where? How can I correct it? What am I missing/not getting?
If my approach is reasonable, then what options are there for validating and choosing a model where both reasonable prediction accuracy AND feature contribution is important to decision making?
Please be gentle, I am still learning.
Any thoughts/questions are welcome!