Interpreting R regression output with multiple interaction variables

Context

I am exploring how different factors in targeting affect subjects' self-reported likeliness to purchase a product. Likeliness to purchase was measured on a four point scale: "Very unlikely", "Unlikely", "Somewhat likely", "Very likely."

I fit two models, the first model compares people's responses to the survey at different times of the day. This model does not take into account my treatment and control variables (whether they had previously been shown an ad or not).

The results are as follows:

glm(formula = as.numeric(gfk_data$AnswerText) ~ gfk_data$daypart)

Deviance Residuals:
Min        1Q    Median        3Q       Max
-2.03341   0.00086   0.00208   0.00376   1.00376

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)                    2.997917   0.028795 104.112   <2e-16 ***
gfk_data$daypartmiddle_of_day 0.001225 0.034217 0.036 0.971 gfk_data$daypartmorning        0.035496   0.036112   0.983    0.326
gfk_data$daypartovernight -0.001681 0.032846 -0.051 0.959 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  So nothing is really significant here. Then what I do is interact the time with the treatment variable so that I can see if people the time of day makes a difference for how likely they were to purchase. The results for that model are: glm(formula = as.numeric(gfk_data$AnswerText) ~ gfk_data$daypart * gfk_data$Exposure)

Coefficients: (1 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept)                                             3.61140    0.21564  16.748  < 2e-16 ***
gfk_data$daypartmiddle_of_day -0.86140 0.38168 -2.257 0.02407 * gfk_data$daypartmorning                                -0.41140    0.35475  -1.160  0.24623
gfk_data$daypartovernight 0.05526 0.04918 1.124 0.26116 gfk_data$Exposurecontrol                               -0.57757    0.21907  -2.636  0.00841 **
gfk_data$Exposureexposed -0.65813 0.21129 -3.115 0.00185 ** gfk_data$daypartmiddle_of_day:gfk_data$Exposurecontrol 0.82915 0.38445 2.157 0.03108 * gfk_data$daypartmorning:gfk_data$Exposurecontrol 0.36258 0.35803 1.013 0.31126 gfk_data$daypartovernight:gfk_data$Exposurecontrol -0.10951 0.06603 -1.659 0.09729 . gfk_data$daypartmiddle_of_day:gfk_data$Exposureexposed 0.90623 0.38024 2.383 0.01720 * gfk_data$daypartmorning:gfk_data$Exposureexposed 0.55103 0.35366 1.558 0.11929 gfk_data$daypartovernight:gfk_data$Exposureexposed NA NA NA NA --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  Question I'm confused because given the base case, which is evening in both models above, how is it that daypartmiddle_of_day:gfk_data$Exposureexposed and daypartmiddle_of_day:gfk_data$Exposurecontrol have both have positive significant coefficients in the interaction model but middle_of_day in in the first model shows very little difference? Am I interpreting this correctly in saying that both daypartmiddle_of_day:gfk_data$Exposureexposed and daypartmiddle_of_day:gfk_data$Exposurecontrol have on average ratings that are .8-.9 higher than the base case, which is the aggregate of control and exposed for evening? • As a side note, I realized I made somewhat of a logical mistake in assembling the dataset I'm using for this model. What I really want to measure is if the time of the impression has an impact on the self reported likeliness to purchase... but regardless, the statistics question here still stands – mdiscenza Jul 21 '16 at 22:30 • When you have complex model output, it's incredibly helpful to plot your model predictions. The 'effects' package makes this very simple and I would recommend trying it out. – mkt Jul 22 '16 at 7:19 1 Answer There are two things that strike me. Let me demonstrate a little R code I ran to reproduce your dataset: set.seed(10) score <- sample(c(0,1,2,3), 40, replace = T) daypart <- sample(c("Morning", "Midday", "Evening", "Night"), 40, prob = c(1/3, 1/3, 1/6, 1/6), replace = T) exposure <- sample(c("Control", "Exposed"), 40, replace = T) summary(glm(score ~ daypart)) Deviance Residuals: Min 1Q Median 3Q Max -1.5294 -0.5294 -0.2059 0.5882 1.5882 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 1.6667 0.6011 2.773 0.00875 ** daypartMidday -0.1373 0.6520 -0.211 0.83444 daypartMorning -0.2549 0.6520 -0.391 0.69812 daypartNight 0.3333 0.8501 0.392 0.69727  So, as in your dataset, no time of day is doing significantly "better" than Evening. (At this point another comment becomes necessary: By doing the analysis your way, you only compare Midday, Morning and Night AGAINST Evening. If you were interested in seeing whether there is an overall difference, you would have to run an ANOVA (or in the case of Treatment as Covariate an ANCOVA). If you were interested in contrasts of the different times of the day, you could run Tukey.) Back to the dataset. So far no time of day was significantly better than Evening. So now we will include the treatment in our model: summary(glm(score ~ daypart*exposure)) Deviance Residuals: Min 1Q Median 3Q Max -1.6250 -0.6250 0.0000 0.6111 1.7778 Coefficients: (1 not defined because of singularities) Estimate Std. Error t value Pr(>|t|) (Intercept) 1.000e+00 1.066e+00 0.938 0.355 daypartMidday 4.444e-01 1.124e+00 0.395 0.695 daypartMorning 2.222e-01 1.124e+00 0.198 0.844 daypartNight 8.158e-16 9.735e-01 0.000 1.000 exposureExposed 1.000e+00 1.306e+00 0.766 0.449 daypartMidday:exposureExposed -8.194e-01 1.405e+00 -0.583 0.564 daypartMorning:exposureExposed -5.972e-01 1.405e+00 -0.425 0.674 daypartNight:exposureExposed NA NA NA NA  1) As you can see (the same as in your model provided above) you get the warning: "(1 not defined because of singularities)", which means there is singularity in your data. The problem (in my case, I assume for you it will be the same) is quickly found: sum(daypart == "Night" & exposure == "Control") [1] 0  So there are no observations of "Night" and "Control" in the dataset. You will have a similar problem. 2) As you can see in my example, my Intercept contains "Evening" and "Control", so every parameter estimate is interpretable. e.g. Individuals who were both exposed and shopped in the morning will on average display a 0.59 lower score than my Intercept, so Individuals who shop in the Evening and are not exposed (this effect, however, is not significant). In your data provided, it is unclear, what the Intercept is. Obviously it contains "Evening", but what is aggregate of control and exposed ? If it is the mean of the two, then how can both Control and Exposure have a negative effect? Also this seems a bit over complicated for me, it would be much easier and cleaner to just compare control to exposure. So to sum it up: Firstly, I suggest you take a look at how your "gfk_data$Exposure" is defined and I suggest coding it just with 0/1 or C/E. Then you will get cleaner interpretations of your parameters. Secondly, I suggest you rethink whether a glm is actually what you want to use here. (AN(C)OVA, Tukey, ...), since it's p-values will only allow you to interpret significant differences towards the reference time&treatment, and not overall. If you now run your analysis again, I am sure the results will be less confusing.

• Ah, yes you're right- the graphing helped a lot. I went through and changed the contrasts setting and it made sense after. It basically wasn't worth fitting the model to begin with. – mdiscenza Jul 27 '16 at 14:30
• glad everything resolved after all :) – E L M Jul 27 '16 at 14:56