# Why does Type II/III ANOVA on cumulative link model (R-package ordinal) give different results depending on factor order?

From what I understand, type II and type III ANOVA should give the same result irrespective of the order of the factors in the formula because they calculate:

Type II:
SS(A | B) for factor A.
SS(B | A) for factor B.
SS(A:B|A,B) for the interaction.
Type III:
SS(A | B, A:B) for factor A.
SS(B | A, A:B) for factor B.
SS(A:B|A,B) for the interaction.


But when I run clm from the ordinal R-package on my data and then perform type II (or type III) ANOVA using the Anova function from the car R-package, I obtain completely different results depending on the order of my factors.

Edit 2: After a little bit of trial and error, I managed to make better example data for my issue:

dat<-data.frame(Treatment=c("Control","TreatmentA","TreatmentB","Control","TreatmentA","TreatmentB"),
Judge=c("C","C","C","E","E","E"),
VeryLow=c(2,3,2,2,3,2),
Low=c(1,3,6,1,3,6),
High=c(3,3,5,3,3,5),
VeryHigh=c(68,76,72,68,76,72))

# Some code to make the count data compatible with the clm package
orderedLevels<-c("VeryLow","Low","High","VeryHigh")
longDat<-reshape(dat, varying=orderedLevels, v.names="rating_count",timevar="rating",times=orderedLevels,direction="long")
indivDat<-longDat[rep(seq(1, nrow(longDat)), longDat$rating_count),] indivDat$rating<-ordered(indivDat$rating, levels=orderedLevels) summary(indivDat) # The clm models without interactions library(ordinal);library(car) fm1 <- clm(rating ~ Treatment + Judge, data=indivDat) fm2 <- clm(rating ~ Judge + Treatment, data=indivDat) Anova(fm1, type="II"); Anova(fm2, type="II") # The lm equivalent models without interactions indivDat$rating_num<- as.numeric(indivDat\$rating)
fm3 <- lm(rating_num ~ Treatment + Judge, data=indivDat)
fm4 <- lm(rating_num ~ Judge + Treatment, data=indivDat)
Anova(fm3, type="II"); Anova(fm4, type="II")


The results for this new example are:

> Anova(fm1, type="II")
Analysis of Deviance Table (Type II tests)

Response: rating
Df  Chisq Pr(>Chisq)
Treatment  2 92.725    < 2e-16 ***
Judge      1 53.275    2.9e-13 ***
---

> Anova(fm2, type="II")
Analysis of Deviance Table (Type II tests)

Response: rating
Df  Chisq Pr(>Chisq)
Judge      1 92.426  < 2.2e-16 ***
Treatment  2 76.882  < 2.2e-16 ***
---

> Anova(fm3, type="II")
Anova Table (Type II tests)

Response: rating_num
Sum Sq  Df F value Pr(>F)
Treatment   1.177   2  1.3917 0.2497
Judge       0.000   1  0.0000 1.0000
Residuals 204.659 484

> Anova(fm4, type="II")
Anova Table (Type II tests)

Response: rating_num
Sum Sq  Df F value Pr(>F)
Judge       0.000   1  0.0000 1.0000
Treatment   1.177   2  1.3917 0.2497
Residuals 204.659 484


How can the Judge effect be significant with the clm package when the data for each judge is just a copy of each other? And why does the order of the factor in the cum link model matters?

What is going on here? Did I misunderstood the type II/III Anova? And what is the best course of action when analyzing such a data set?

Original example (without results): Here is a reproducible example for Type II using the wine dataset:

library(ordinal)
fm1 <- clm(rating ~ contact * bottle, data=wine)
fm2 <- clm(rating ~ bottle * contact, data=wine)

library(car)
Anova(fm1, type="II"); Anova(fm2, type="II")


Edit 1: Same model without interactions.

> fm1 <- clm(rating ~ contact + bottle, data=wine)
> Anova(fm1, type="II")
Analysis of Deviance Table (Type II tests)

Response: rating
Df   Chisq Pr(>Chisq)
contact  1  1.8182     0.1775
bottle   7 53.1100  3.526e-09 ***

> fm2 <- clm(rating ~ bottle + contact, data=wine)
> Anova(fm2, type="II")
Analysis of Deviance Table (Type II tests)

Response: rating
Df  Chisq Pr(>Chisq)
bottle   7 71.534  7.231e-13 ***
contact  1  6.847   0.008879 **

• You've got a problem with your data. The df for the interaction should be 7. Given that it is 3 indicates that you have cells with no data. – dbwilson Sep 28 '17 at 13:31
• wine is the example data from the ordinal package. It does not contain any missing data. @dbwilson You are right that the interaction doesn't make sense here because bottles 1,2,5 and 6 always have "no" contact and the rest of the bottles (3,4,7 and 8) always have a "yes" for contact. But (1) removing the interaction term does not solve the problem. (2) Shouldn't Anova type II and III results be independent of the order of the factor even if the data is unbalanced? – Prolix Sep 28 '17 at 13:45
• That is the problem. Contact is fully determined by bottle. As such, this data is not appropriate for a two-way ANOVA. With data in all 16 cells (8 bottles, 2 contacts) then the order of the factors would not matter. Also note that this is not a standard ANOVA model (I didn't notice this initially) given that the DV is coded as ordinal and not scaled (hence the analysis of deviance with a chisq rather than analysis of variance with an F). I'm not sure how the latter would be affected by the Type II/III distinction. – dbwilson Sep 28 '17 at 14:02
• @dbwilson (1) If the data was perfectly balanced, not only would both order give the same results but ANOVA type I, II and III would also give the same results. So that would completely defeat the purpose of this question. Of course this is an extreme example, but I get the same with less unbalanced data. (2) Indeed, I only get this issue with clm models for ordinal response. But shouldn't the Chi-square test be symmetrical just like the F-test? Otherwise, what does Type II and III mean in for this test? – Prolix Sep 28 '17 at 14:52
• In your example, bottles are nested within contacts. Correcting what I said above, you can do a two-way anova but the interaction between these two doesn't make sense: it is not possible for the effect of bottle to differ between contacts=0 and contacts=1. – dbwilson Sep 28 '17 at 19:02

The documentation for car::Anova does not indicate that it works with clm or clmm objects. From your examples, it appears it can have some unusual behavior.

My solution is to use RVAideMemoire::Anova.clm.

So, for your EDIT 2 example:

if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}
if(!require(car)){install.packages("car")}

library(car)

library(RVAideMemoire)

Anova(fm1, type="II")

### Analysis of Deviance Table (Type II tests)

### Response: rating
###           LR Chisq Df Pr(>Chisq)
### Treatment   4.0507  2     0.1319
### Judge       0.0000  1     1.0000

Anova(fm2, type="II")

### Analysis of Deviance Table (Type II tests)

### Response: rating
###           LR Chisq Df Pr(>Chisq)
### Judge       0.0000  1     1.0000
### Treatment   4.0507  2     0.1319


Note that this produces the same p-values as the anova function.

fm1 <- clm(rating ~ Treatment + Judge, data=indivDat)
fmx <- clm(rating ~ Treatment, data=indivDat)
fmy <- clm(rating ~ Judge, data=indivDat)

anova(fm1, fmx)

### Likelihood ratio tests of cumulative link models:

### fmx rating ~ Treatment         logit flexible
### fm1 rating ~ Treatment + Judge logit flexible

###     no.par    AIC  logLik LR.stat df Pr(>Chisq)
### fmx      5 474.85 -232.42
### fm1      6 476.85 -232.42       0  1

anova(fm1, fmy)

### Likelihood ratio tests of cumulative link models:


• Thank you a lot! These results make much more sense, and it is a bit simpler to use than the "manual" comparison of models with anova, especially for models with more predictors! It would be nice if car::Anova would give an error or a warning of some sort for 'clm' objects... – Prolix Oct 3 '17 at 10:21
• If I type if(!require(RVAideMemoire)){install.packages("RVAideMemoire")}; library(RVAideMemoire); Anova(fm1, type="II"), I get Error in Anova(fm1, type = "II") : could not find function "Anova"! Would you know what is going on? – Prolix Oct 4 '17 at 15:46
• You need a library(car) before the library(RVAideMemoire). Let me change that in my answer. – Sal Mangiafico Oct 4 '17 at 17:12