I'm trying to assess the effect of showing more impressions on a user. I want to study if users who saw more ads are more likely to make a purchase onsite. To do so I've created a multilevel model. I grouped users into 10 groups averaging their scores (we score users based on a number of factors). So basically I end up having 10 groups (from 0 to 9), where on group 9 I assume to have the best users, and on group 0 the worst.

  picbucket mcuserid impressions mediacostcpm is_buyer gr.impressions gr.mediacostcpm
1         0 1           1        0.460        0       3.632794        2.767509
2         0 2           2        5.000        0       3.632794        2.767509
3         0 3           1        4.590        0       3.632794        2.767509
4         0 4           1        0.590        0       3.632794        2.767509
5         0 5           1        5.000        0       3.632794        2.767509
6         0 6           1        0.315        0       3.632794        2.767509

I think a multilevel model could be advantageous here because I'm expecting to see different effects on each group. On the best users I'm expecting an additional impression could have a higher impact, whereas on bad users and additional impression may be worthless. It could also be possible the opposite though. So that users which generally higher score will convert even without the need of serving them more impressions, whereas on mid groups additional impressions tend to change their behaviour.

A good model representation could be:

$y_{i} = \alpha_{j[i]} + X_{i}\beta + \epsilon_{i}$

The second level of the model will then be:

$\alpha_{j} = \mu_{\alpha} + \eta_{j}, \text{ with } \eta_{j} \sim N(0, \sigma_{\alpha}^{2})$

On the first level I want to include as a predictor how many impression a user saw. On the second level I want to include the average impressions a user saw within its group and the average media cost for the impressions we served on that user. I've used the package lme4 in R to build my model.

glmer(formula = is_buyer ~ impressions + mediacostcpm + (1 + 
    gr.impressions + gr.mediacostcpm | picbucket), data = new.df, 
    family = binomial())
             coef.est coef.se
(Intercept)  -7.42     0.33  
impressions   0.00     0.02  
mediacostcpm  0.03     0.01  

Error terms:
 Groups    Name            Std.Dev. Corr        
 picbucket (Intercept)     7.86                 
           gr.impressions  2.22     -0.99       
           gr.mediacostcpm 0.57     -0.68  0.60 
 Residual                  1.00                 
number of obs: 103146, groups: picbucket, 10
AIC = 2755.4, DIC = 2680.5
deviance = 2708.9 

This is my first experiment with multilevel modeling so I would like to make sure I don't misunderstand the results of my model.

From what I see here, the impressions predictor on the first level is useless. Its coefficient is zero and its standard deviation is very small. This could be due to the fact I'm including a group average on the second level for the impression count (gr.impressions). So, on any group (picbucket), serving more impressions than the average doesn't tell us much about the likelihood of a cookie to convert.

The average media cost on the first level is however an interesting one. Generally, within a group, if I spend a bit more for every impression I should increase the probability of generating conversions. This is probably due to inventory quality. Better inventory costs more, but also has better changes to be viewable inventory.

The coefficients on the group level instead tell you how much they contribute on explaining the group slope. So in this case the average number of impressions at the group level seems to explain quite a significant part of the group slope.

Interestingly, at the group level gr.impressions seem to be a very useful predictor, but at the within group level its usefulness is limited. The opposite applies to the mediacostcpm.

Am I interpreting these results correctly? How can I tell if the model has a good fit? Please note I've used a binomial regression because the dependent variable, is_buyer can take only 0 or 1 (one being the user made a purchase).



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.