0
$\begingroup$

Context

I am exploring how different factors in targeting affect subjects' self-reported likeliness to purchase a product. Likeliness to purchase was measured on a four point scale: "Very unlikely", "Unlikely", "Somewhat likely", "Very likely."

I fit two models, the first model compares people's responses to the survey at different times of the day. This model does not take into account my treatment and control variables (whether they had previously been shown an ad or not).

The results are as follows:

glm(formula = as.numeric(gfk_data$AnswerText) ~ gfk_data$daypart)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-2.03341   0.00086   0.00208   0.00376   1.00376  

Coefficients:
                               Estimate Std. Error t value Pr(>|t|)    
(Intercept)                    2.997917   0.028795 104.112   <2e-16 ***
gfk_data$daypartmiddle_of_day  0.001225   0.034217   0.036    0.971    
gfk_data$daypartmorning        0.035496   0.036112   0.983    0.326    
gfk_data$daypartovernight     -0.001681   0.032846  -0.051    0.959    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

So nothing is really significant here.

Then what I do is interact the time with the treatment variable so that I can see if people the time of day makes a difference for how likely they were to purchase.

The results for that model are:

glm(formula = as.numeric(gfk_data$AnswerText) ~ gfk_data$daypart * 
    gfk_data$Exposure)

    Coefficients: (1 not defined because of singularities)
                                                       Estimate Std. Error t value Pr(>|t|)    
(Intercept)                                             3.61140    0.21564  16.748  < 2e-16 ***
gfk_data$daypartmiddle_of_day                          -0.86140    0.38168  -2.257  0.02407 *  
gfk_data$daypartmorning                                -0.41140    0.35475  -1.160  0.24623    
gfk_data$daypartovernight                               0.05526    0.04918   1.124  0.26116    
gfk_data$Exposurecontrol                               -0.57757    0.21907  -2.636  0.00841 ** 
gfk_data$Exposureexposed                               -0.65813    0.21129  -3.115  0.00185 ** 
gfk_data$daypartmiddle_of_day:gfk_data$Exposurecontrol  0.82915    0.38445   2.157  0.03108 *  
gfk_data$daypartmorning:gfk_data$Exposurecontrol        0.36258    0.35803   1.013  0.31126    
gfk_data$daypartovernight:gfk_data$Exposurecontrol     -0.10951    0.06603  -1.659  0.09729 .  
gfk_data$daypartmiddle_of_day:gfk_data$Exposureexposed  0.90623    0.38024   2.383  0.01720 *  
gfk_data$daypartmorning:gfk_data$Exposureexposed        0.55103    0.35366   1.558  0.11929    
gfk_data$daypartovernight:gfk_data$Exposureexposed           NA         NA      NA       NA    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Question I'm confused because given the base case, which is evening in both models above, how is it that daypartmiddle_of_day:gfk_data$Exposureexposed and daypartmiddle_of_day:gfk_data$Exposurecontrol have both have positive significant coefficients in the interaction model but middle_of_day in in the first model shows very little difference?

Am I interpreting this correctly in saying that both daypartmiddle_of_day:gfk_data$Exposureexposed and daypartmiddle_of_day:gfk_data$Exposurecontrol have on average ratings that are .8-.9 higher than the base case, which is the aggregate of control and exposed for evening?

$\endgroup$
  • $\begingroup$ As a side note, I realized I made somewhat of a logical mistake in assembling the dataset I'm using for this model. What I really want to measure is if the time of the impression has an impact on the self reported likeliness to purchase... but regardless, the statistics question here still stands $\endgroup$ – mdiscenza Jul 21 '16 at 22:30
  • 3
    $\begingroup$ When you have complex model output, it's incredibly helpful to plot your model predictions. The 'effects' package makes this very simple and I would recommend trying it out. $\endgroup$ – mkt Jul 22 '16 at 7:19
3
$\begingroup$

There are two things that strike me. Let me demonstrate a little R code I ran to reproduce your dataset:

set.seed(10)

score <- sample(c(0,1,2,3), 40, replace = T)
daypart <- sample(c("Morning", "Midday", "Evening", "Night"), 40, prob =  c(1/3, 1/3, 1/6, 1/6), replace = T)
exposure <- sample(c("Control", "Exposed"), 40, replace = T)

summary(glm(score ~ daypart))

Deviance Residuals: 
Min       1Q   Median       3Q      Max  
-1.5294  -0.5294  -0.2059   0.5882   1.5882  

Coefficients:
           Estimate Std. Error t value Pr(>|t|)   
(Intercept)      1.6667     0.6011   2.773  0.00875 **
daypartMidday   -0.1373     0.6520  -0.211  0.83444   
daypartMorning  -0.2549     0.6520  -0.391  0.69812   
daypartNight     0.3333     0.8501   0.392  0.69727   

So, as in your dataset, no time of day is doing significantly "better" than Evening. (At this point another comment becomes necessary: By doing the analysis your way, you only compare Midday, Morning and Night AGAINST Evening. If you were interested in seeing whether there is an overall difference, you would have to run an ANOVA (or in the case of Treatment as Covariate an ANCOVA). If you were interested in contrasts of the different times of the day, you could run Tukey.)

Back to the dataset. So far no time of day was significantly better than Evening. So now we will include the treatment in our model:

summary(glm(score ~ daypart*exposure))

Deviance Residuals: 
Min       1Q   Median       3Q      Max  
-1.6250  -0.6250   0.0000   0.6111   1.7778  

Coefficients: (1 not defined because of singularities)
                             Estimate Std. Error t value Pr(>|t|)
(Intercept)                     1.000e+00  1.066e+00   0.938    0.355
daypartMidday                   4.444e-01  1.124e+00   0.395    0.695
daypartMorning                  2.222e-01  1.124e+00   0.198    0.844
daypartNight                    8.158e-16  9.735e-01   0.000    1.000
exposureExposed                 1.000e+00  1.306e+00   0.766    0.449
daypartMidday:exposureExposed  -8.194e-01  1.405e+00  -0.583    0.564
daypartMorning:exposureExposed -5.972e-01  1.405e+00  -0.425    0.674
daypartNight:exposureExposed           NA         NA      NA       NA

1) As you can see (the same as in your model provided above) you get the warning: "(1 not defined because of singularities)", which means there is singularity in your data. The problem (in my case, I assume for you it will be the same) is quickly found:

sum(daypart == "Night" & exposure == "Control")
[1] 0 

So there are no observations of "Night" and "Control" in the dataset. You will have a similar problem.

2) As you can see in my example, my Intercept contains "Evening" and "Control", so every parameter estimate is interpretable. e.g. Individuals who were both exposed and shopped in the morning will on average display a 0.59 lower score than my Intercept, so Individuals who shop in the Evening and are not exposed (this effect, however, is not significant).

In your data provided, it is unclear, what the Intercept is. Obviously it contains "Evening", but what is

aggregate of control and exposed

? If it is the mean of the two, then how can both Control and Exposure have a negative effect? Also this seems a bit over complicated for me, it would be much easier and cleaner to just compare control to exposure.

So to sum it up: Firstly, I suggest you take a look at how your "gfk_data$Exposure" is defined and I suggest coding it just with 0/1 or C/E. Then you will get cleaner interpretations of your parameters. Secondly, I suggest you rethink whether a glm is actually what you want to use here. (AN(C)OVA, Tukey, ...), since it's p-values will only allow you to interpret significant differences towards the reference time&treatment, and not overall. If you now run your analysis again, I am sure the results will be less confusing.

$\endgroup$
  • $\begingroup$ Ah, yes you're right- the graphing helped a lot. I went through and changed the contrasts setting and it made sense after. It basically wasn't worth fitting the model to begin with. $\endgroup$ – mdiscenza Jul 27 '16 at 14:30
  • $\begingroup$ glad everything resolved after all :) $\endgroup$ – E L M Jul 27 '16 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.