Mixed-effects models with customer data: how do choices affect the model?

Suppose I had a large sample of customer data from which I want to predict total amount of sales over a time period with predictor variables indicating:

-which sales channel did customers come from (e.g. internet, phone, store), which has a big impact on the variable to predict and the categories are not going to change.

-demographic data (e.g. age, gender, address) including zip code.

-detailed information for each zip code (e.g. average income, proportion of people with university degrees, etc.).

And I wanted to use them to both predict and make inferences on the total amount purchased over a period of time (not necessarily having the same model for both).

-Using a nested structure of state/city/zip code vs. appending the aggregated information from the zip code to each record (i.e. replacing zip code by columns indicating average income and such in that specific zip code).

-Treating the sales channel as a random effect vs. a fixed effect interacting with others, in terms of predictive accuracy.

Finally, how would you decide on classifying the sales channel as either a fixed or random effect?

Sales channel should be fixed because:

a) the list is exhaustive. The sales channels in your sample cover all possible sales channels.

b) You are not trying to infer to other, as yet unseen, sales channels.

c) The number of sales channels is small. It's hard to estimate the variance of a fixed effect without a reasonable number of levels - say 10 or more, and I prefer to see more.

The mixed effect model would assume that the sales channel effect is normally distributed -- in other words, it sort of randomly adjusts the outcome, but in symmetric and controlled ways. Suppose that phone purchases are WAY lower than purchases made in stores and over the Internet because people can't browse on the phone and make impulse purchases. A mixed model would tend to adjust the impact of phone sales upwards to make them more typical of other channels. Whereas with fixed effects, the outlier status of phone sales would be highlighted.

Anyway, I would certainly use fixed effects in your situation if you only have the 3 sales channels that you mention.

The thing about predicting mixed effects models is that they predict the mean of future observations on the basis of the fixed effects, but adjust the size of the confidence intervals to take into account the additional variability caused by the random effects. You can estimate the value of the random effects for the data you have ... so you could estimate the purchasing effect of zip code 12345, say, if it's in your sample. The effect would be shrunk towards zero, in comparison to the fixed effect you would have from treating zip code as a fixed effect. But if you wanted to predict sales from an unseen zip code, the predicted mean would depend on the fixed effects alone. However, if you included average income, and knew the average income of some unseen zip neighbourhood, you could use that information to refine your prediction.

So good idea to include average income.

• Thanks! now, just to make sure (I'm just trying to understand how random-effects work and I got found contradictory info googleing it), if I go to R and type something like this: lmer(y~1+ (1|a)), then 'a' will have no impact when I call predict on new data? Jul 29 '15 at 17:49
• Presumably you don't know the value of $a$ when you are doing the prediction. Suppose y is a child's score on a standardized math test and $a$ is the school (random effect). When I want to predict the score of an average child from an average school, I don't know the school or its effect on math learning. The random effect impacts the variance of the prediction, however. Jul 29 '15 at 17:58
• If you check the documentation of predict.merMod, you can see that prediction depends on whether you want to predict for levels you have not previously seen. When you are predicting for known levels, it uses the estimated value of the random effects at that level. If not, then not. Jul 29 '15 at 18:27

(Posted this as a follow-up due to the size) Ok, but there’s still something I don’t understand. Let’s say I do the following senseless data simulation:

y=rnorm(100,3)
a=abs(floor(abs(y))-1)
s=data.frame(y,a)
s$a=as.factor(a) q=s[1:70,] w=s[71:100,] library(lme4) m1=lm(y~1,data=q) m2=lmer(y~1+(1|a),data=q) m3=lm(y~a,data=q) p1=predict(m1,w) p2=predict(m2,w) p3=predict(m3,w) summary(p1) Min. 1st Qu. Median Mean 3rd Qu. Max. 2.787 2.787 2.787 2.787 2.787 2.787 summary(p2) Min. 1st Qu. Median Mean 3rd Qu. Max. 1.621 2.377 3.503 3.172 3.503 5.015 summary(p3) Min. 1st Qu. Median Mean 3rd Qu. Max. 1.593 2.373 3.504 3.182 3.504 5.264 summary(w$y)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.529   2.539   3.132   3.190   3.959   5.128
cor(p2,w$y)  0.9443712 cor(p3,w$y)
 0.9464884

In this case, treating ‘a’ as a fixed or random effect did have a noticeable difference in the predictions, and the intercept coefficient is different in the null model and the mixed-effect model.

summary(m2)
Linear mixed model fit by REML ['lmerMod']
Formula: y ~ 1 + (1 | a)
Data: q

REML criterion at convergence: 53.9

Scaled residuals:
Min       1Q   Median       3Q      Max
-1.85541 -0.88258  0.07429  0.75298  2.09967

Random effects:
Groups   Name        Variance Std.Dev.
a        (Intercept) 3.21934  1.7943
Residual             0.08187  0.2861
Number of obs: 70, groups:  a, 6

Fixed effects:
Estimate Std. Error t value
(Intercept)   3.9108     0.7361   5.313

summary(m1)

Call:
lm(formula = y ~ 1, data = q)

Residuals:
Min      1Q  Median      3Q     Max
-2.0065 -0.6203  0.0062  0.3468  3.2418

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)   3.0612     0.1137   26.93   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.951 on 69 degrees of freedom

coef(m2)
\$a
(Intercept)
0    1.499427
1    2.543697
2    3.365850
3    4.388612
4    5.423515
5    6.243746

coef(m3)
(Intercept)          a1          a2          a3          a4          a5
1.490667    1.051693    1.874669    2.899463    3.971315    4.812404

What would the 3.9108 from the mixed-effects model represent? Is it like a more robust estimation of the intercept?

In which kind of situations would you build a mixed-effects model to make predictions?