3
$\begingroup$

I am trying to get my head around the difference between a fixed effect versus a random effect. To do this, I am looking at the Student GPA example seen here (from m-clark.github). This example is "assessing the factors that predict college grade point average (GPA). Each of 200 students are assessed for six occasions (each semester for the first three years), so we have observations clustered within students."

Note that the data for this example can be downloaded here.

The example starts by fitting a standard linear regression, with only occasion as a fixed effect.

load('data/gpa.RData')
gpa_lm = lm(gpa ~ occasion, data = gpa)
summary(gpa_lm)

This produces the following output:

Call:
lm(formula = gpa ~ occasion, data = gpa)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.90553 -0.22447 -0.01184  0.26921  1.19447 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 2.599214   0.017846  145.65   <2e-16 ***
occasion    0.106314   0.005894   18.04   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.3487 on 1198 degrees of freedom
Multiple R-squared:  0.2136,    Adjusted R-squared:  0.2129 
F-statistic: 325.3 on 1 and 1198 DF,  p-value: < 2.2e-16

Subsequently, the example demonstrates the fitting of a mixed effects model, where occasion is again included as a fixed effect, but this time student is included as a random (intercept) effect:

library(lme4)
gpa_mixed = lmer(gpa ~ occasion + (1 | student), data = gpa)
summary(gpa_mixed)

Producing the following output:

Linear mixed model fit by REML ['lmerMod']
Formula: gpa ~ occasion + (1 | student)
   Data: gpa

REML criterion at convergence: 408.9

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.6169 -0.6373 -0.0004  0.6361  2.8310 

Random effects:
 Groups   Name        Variance Std.Dev.
 student  (Intercept) 0.06372  0.2524  
 Residual             0.05809  0.2410  
Number of obs: 1200, groups:  student, 200

Fixed effects:
            Estimate Std. Error t value
(Intercept) 2.599214   0.021696   119.8
occasion    0.106314   0.004074    26.1

Correlation of Fixed Effects:
         (Intr)
occasion -0.469

Now I can clearly see that in both models, the estimate for the global intercept remains the same (2.599). And I understand that in the mixed effects model each student gets their own intercept estimate; I can see these by running the following code:

coef(gpa_mixed)$student[1:5,]

Resulting in the following output (only the first 5 students shown):

  (Intercept)  occasion
1    2.528319 0.1063143
2    2.383636 0.1063143
3    2.687471 0.1063143
4    2.412573 0.1063143
5    2.629598 0.1063143

My question, hence, is what is the difference between the above mixed effects model, which includes student as a random effect, versus a fixed effects model that includes student as a fixed effect?

In both cases an intercept effect is estimated for each student, so what is it about the two models that is different?

To exemplify, here is the fixed effects model:

gpa_fixed = lm(gpa ~ occasion + student, data = gpa)
summary(gpa_fixed)

This model produces the following summary – note that again only students 1-5 are shown, with the estimate for student 1 being represented by "(Intercept)":

Call:
lm(formula = gpa ~ occasion + student, data = gpa)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.8676 -0.1474  0.0072  0.1438  0.7325 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.518e+00  9.892e-02  25.450  < 2e-16 ***
occasion     1.063e-01  4.074e-03  26.096  < 2e-16 ***
student2    -1.667e-01  1.392e-01  -1.198 0.231301    
student3     1.833e-01  1.392e-01   1.318 0.187967    
student4    -1.333e-01  1.392e-01  -0.958 0.338196    
student5     1.167e-01  1.392e-01   0.838 0.401995  

From this I can see that the estimate for the slope effect of occasion remains unchanged (0.10634 in both the fixed effects and the mixed effects model), but the estimates for the intercept effect for each student are slightly different.

For example, in the mixed effects model, student 1 has an intercept estimate of 2.528319, whereas in the fixed effects model the intercept estimate for student 1 is 2.518.

Similarly, in the mixed effects model, student 2 has an intercept estimate of 2.383636, whereas in the fixed effects model the intercept estimate for student 2 is 2.518-0.1667=2.3513.

Admittedly these values are very similar, but they are not identical.

Are these values different only because of the way that they are estimated? i.e. partial pooling/shrinkage in the case of the mixed effects model versus least-squares in the fixed effects model?

My understanding is that when you include student as a random effect, the estimate for each student's intercept is drawn from a normal distribution centered on the global intercept estimate. Whereas when you include student as a fixed effect, each student's intercept can be estimated with whatever value is best (i.e. best according to least squares) without any constraint as to what distribution that estimate is drawn from. Is my understanding here correct?

Why would it matter, from a statistical point of view, if I modeled student as a fixed effect, rather than as a random effect? Is it for example because with the fixed effect you cannot "borrow strength" from other groups? And because with the fixed effect you are having to estimate a greater number parameters than with the random effect, which only considers variance around the global intercept estimate?

Thank you in advance!

Edit: I also have another question - how exactly do you interpret the value of the random effect?

Random effects:
 Groups   Name        Variance Std.Dev.
 student  (Intercept) 0.06372  0.2524  
 Residual             0.05809  0.2410  
Number of obs: 1200, groups:  student, 200

The value for the variance effect size is 0.06372. Does this mean that, when variance of all the individual student intercept estimates is 0.06372? I tried that using var(ranef(gpa_mixed)$student) but this returns a value of 0.0553. If anyone could shed some light on how to intuitively interpret the random effect estimates, that would really help!

$\endgroup$

3 Answers 3

1
$\begingroup$

You are mainly on the right track. We don’t include student as a fixed effect because there are too many of them, requiring too many parameters thus making the model unstable. Random effects, when given the opportunity to have a finite variance, will be shrunken towards the grand mean as you have indicated, the discounting implying that the effective degrees of freedom for student is less than the number of students. Random intercepts also induce a compound symmetric (exchangeable) correlation structure within student.

However the example solution you are reproducing does not represent best statistical practice, for the following reasons:

  1. No diagnostics were included to check that the compound symmetric correlation pattern (which can fit OK if the time span is short) actually fits the data. A variagram should be drawn as in this case study. Serial correlation models typically fit better than random effect models, and they don’t need the complexity of random effects (student would be omitted from the mean model but not from the correlation structure).
  2. It is unlikely that occasion has a linear effect.
  3. The normality of the ordinary residuals wasn’t checked.

It is also possible that the random effects are not normal.

$\endgroup$
0
$\begingroup$

This may require a long explanation but I can point you towards one blog post that has references to the relevant books. In a nutshell, if fitted for just numerical reasons, then a random-effects-model is useful when sample sizes are small, as information sharing and shrinkage happens - less overfitting and all that. If sample sizes are large then it does not make any practical difference in most cases when one considers the added complexity of the model.

From a conceptual or philosophical view point, a random-effect model can be used to estimate 'super-population' parameters.

GELMAN, A., & HILL, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge, Cambridge University Press.

lme4: Mixed-effects modelling with R. Douglas M. Bates (2010)

Kruschke, J. K. (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan, second edition.

https://laplacebayes.wordpress.com/2019/01/04/partial-complete-or-no-pooling-information-content-sample-sizes-and-shrinkage-in-multilevel-regression-models/

I hope this is useful.

Cheers

$\endgroup$
0
$\begingroup$

Some notes about your question:

  • In a mixed effects model the mixed effects are not explicitly computed and they are considered as nuisance parameters.

    Intuition about parameter estimation in mixed models (variance parameters vs. conditional modes)

  • The model is estimated by integrating out the random effects. An example question where this is demonstrated with a manual computation is:

    Why do fixed effects in a logistic regression model differ depending on the presence of a random intercept?

    (this question is also relating to your case where the intercept is either 2.599 or 2.518 depending on using random or fixed effects)

  • The random effects computed by a mixed effects model are an optimization of random effects that is performed after the model is fitted. It answers the question "Given the fixed effects and estimated variances, what are the most likely residuals?".

    In comparison to fixed effects these random effects will be smaller because the assumption of the random effects following a normal distribution will act effectively as a prior on the effect sizes that puts more weight on effects that are close together.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.