Predictive Model for Attribution Model

Question

There is a video on youtube for a talk on use of GBM for building a marketing attribution model at netflix. Essentially, this is a binary classification problem where a visitor to the site either converts or does not (1 or 0) and there are assorted information about the visitor (e.g. what other ads they clicked on and how long ago, if they got an email from the company etc). I believe the goal is to use the model to say of all the conversions we got, how many should be attributed to different things that happened to the customer. So, the company can determine if sending an email really matters for example or how important was Ad 123 for conversion.

My question is how would the fitted model be used (completely skipped in the video).

Here is one way I was thinking about - does it make sense or are there better ones?

Define 'touch-points' as channels, ad types etc => whatever you want to attribute conversions to which have a cost (other things would be controls in the model to adjust for other circumstances like day of the week).

• Run the data through the fitted model and observe the mean probability of conversion given the observed predictors over all visitors: P(conversion=1 | X). This is the baseline.

• For each touch point variable, x_i set the value to zero (and all interactions to zero for example that involve it) for all visitors and run this data through the model. Again, observe the mean probability of conversion: P(conversion=1 | x; x_i=0).

• Calculate the net effect for touch-point i ==> net_effect_i = P(conversion=1 | X) - P(conversion=1 | x; x_i=0)

• Once done for all touch-points, normalize the net effects as follows==> norm_net_effect_i = net_effect_i/sum(net_effect_i for all i).

• Then multiply the normalized net effects and the total true conversions to attribute the number of conversions to each touch-point.

Answer 1

I think they way you are thinking about it is correct, but I would add a little bit more related to how I think the company would use this information.

1.) Maximize your return on investment. You eluded to this concept in your question, but you missed a key concept: The cost of the variable. Let's consider a simple model of two advertising strategies, A and B, which cost 100\$ and 500\$ respectively. If you acquire 30 costumers for A and 60 customers from B who will pay a 10\$ subscription fee, strategy A would yield a return of 200\$ while A would yield 100\$. Thus, it would make more sense to shift resources to A from B, even though B get's you more customers.

2.) Better coordination between services. In point 1, we assumed uncorrelated events. However, it could very well be that the customer bought the product because A occurred before B and not when B occurred before A. Thus, you would also want to account for this in your return on investment. Finding correlations between marketing strategies could yield some very interesting and non-trivial ways to increase the chances of acquiring new customers.

3.) Gaining a better understanding of your customers psychology to create new marketing techniques.. These models are not just numbers, they are representations of complex thought processes across a wide range of individuals. To understand why your customer purchased your product may be the biggest gain from this model.

I'm not really sure if this constitutes an answer, but this is how I would leverage the data if I were in their position. I'm sure there are many other ways as well.

Answer 2

I think the approach you've outlined sounds reasonable, but I suspect it's probably not what they're doing, given that they're using gradient boosting. They're probably extracting normalized feature importances from the gradient boosted trees, like this in this random forest example from scikit-learn (http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html).

Trees are commonly used to identify the most useful/important features. Feature importance is calculated by looking at how much error each variable reduces when it is used in a tree split. For each variable, these reductions are summed over all splits where the variable is used, and then the summed error reductions are normalized by dividing by the sum of the variable with the largest total error reduction.

Trees' variable importances don't indicate direction (good or bad), just relative importance, but that's likely sufficient for Netflix's purposes since it's unlikely that any touch point would have a negative effect.