5
$\begingroup$

I'm wondering how different are generalized linear model and ANOVA. I thought it was because of categorical vs continuous variables, but it doesn't seems to be the case.

Also, I know that it's possible to add mixed effects in a GLMM. Is it the case for ANOVAs?

So, why would one use GLM vs ANOVAs?

Here is an example:

# Difference regression vs ANOVA vs GLM
gr1 = rnorm(10,10,1)
gr2 = rnorm(10,20,1)

df=data.frame(nm = c(gr1,gr2), 
              gr = c(rep(0,length(gr1)),
                     rep(1,length(gr2))))

enter image description here

ANOVA

summary(aov(nm~gr, data = df))

            Df Sum Sq Mean Sq F value  Pr(>F)    
gr           1  496.4   496.4   621.1 2.1e-15 ***
Residuals   18   14.4     0.8                    
---
Signif. codes:  
0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

GLM

glm(nm~gr,family = "gaussian", data = df)

Call:  glm(formula = nm ~ gr, family = "gaussian", data = df)

Coefficients:
(Intercept)           gr  
     10.050        9.964  

Degrees of Freedom: 19 Total (i.e. Null);  18 Residual
Null Deviance:      510.8 
Residual Deviance: 14.39    AIC: 56.17

And once we are there, why not comparing linear regressions and t-test!

Linear regression

summary(lm(nm~gr, data = df))

Residuals:
     Min       1Q   Median       3Q      Max 
-1.13990 -0.85335  0.00061  0.41865  1.92899 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  10.0502     0.2827   35.55  < 2e-16 ***
gr            9.9638     0.3998   24.92  2.1e-15 ***
---
Signif. codes:  
0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.894 on 18 degrees of freedom
Multiple R-squared:  0.9718,    Adjusted R-squared:  0.9703 
F-statistic: 621.1 on 1 and 18 DF,  p-value: 2.095e-15

t-test

tt=t.test(df$nm~df$gr);tt 
    Welch Two Sample t-test

data:  df$nm by df$gr
t = -24.922, df = 17.914, p-value = 2.347e-15
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -10.804012  -9.123526
sample estimates:
mean in group 0 mean in group 1 
       10.05023        20.01400


(tt$statistic)^2
       t 
621.0907 
$\endgroup$
  • $\begingroup$ ANOVA assumes Gaussian distribution of the residuals (and uses a linear model that minimizes the sum of squares, which can be used in a F-statistic). GLM generalizes the linear model used in ANOVA by allowing any other type of distribution of the residuals (and optimizes the likelihood function, which only allows a t-test based on an estimated error of the coefficients). $\endgroup$ – Sextus Empiricus Jul 6 '17 at 23:36
  • 1
    $\begingroup$ So an anova is an Glm, but a Glm is not only anovas. $\endgroup$ – M. Beausoleil Jul 6 '17 at 23:56
  • 1
    $\begingroup$ So an ANOVA is GLM. It is a bit semantic now. 1) GLM includes ordinary least squares. You can't really say OLS is a GLM (since GLM is general and OLS is not). Better would be to say that OLS is GLM with identity link and variance functions. 2) ANOVA is not exactly the same as OLS. It is the analysis performed on a linear regression (on the sum of squares of residuals). Also, ANOVA refers to analysis of reduced var_residuals which you can't do (makes no sense) with GLM. However, in practice, people doing ANOVA often do OLS and model effects. But technically $ANOVA \neq OLS \in GLM$, $\endgroup$ – Sextus Empiricus Jul 7 '17 at 1:56
  • $\begingroup$ The normal linear model is a special case of GLM, while OLS is a distribution free algorithm to find its solution. The classic t-test is a special case of the normal linear model. $\endgroup$ – Michael M Jul 7 '17 at 5:58
  • $\begingroup$ OK, and OLS is a special (as well as ANOVA and t test) case of linear model which is a special case of GLM, $\endgroup$ – M. Beausoleil Jul 7 '17 at 13:32
7
$\begingroup$

This is a quite common point of confusion, as the word ANOVA is used with different meanings in different textbooks / software packages. I'll try to sort this out a bit:

Historically, ANOVA is a method to partition out the contribution of different grouping factors to the variation in a continuous variable. The classical ANOVA methods did so via sum of squares, which in the end corresponds to the assumptions of an lm (iid normal and so on), thus you will often read that AVOVA = normal distribution.

However, this obviously doesn't generalize to GLM(Ms), but people would still like to test for the significance (and contribution) of entire grouping factors in those models, and thus the concept has been extended. A more modern way to look at ANOVA is that an ANOVA is just a series of likelihood ratio tests that add / remove categorical predictors to test for their overall significance in a regression, as well as changes in deviance (acting as a pseudo R2).

So, regardless of lm, glm, glmer, if you have a model with a predictor color = red, green, blue, in R

m1 <- lm/glm/glmer(res ~ color)
summary(m1)

will give you p-values for the contrasts between red, green, blue, while

am1 <- anova(m1)
summary(am1)

will essentially do a likelihood ratio test to decide if the predictor color is supported at all (note that adding color, you are adding 2 df / parameters at once).

In R there are different ANOVA functions (aov, anova, car::ANOVA) that slightly differ in their use and appropriateness for particular regressions and questions. car::ANOVA is very versatile and allows changing between type I, II, II, IV ANOVA (explanation here)

Note that you can also use the anova command in R to do a LRT between two models, as in anova(m1, m2), so in a way you can see anova(m1) simply as a shorthand to compare m1 all of its smaller submodels via a LRTs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.