I am running an ordinal model on rating data, with 1 random effect (subject) and 2 fixed effects (condition with 3 levels, probe.position with 6). I use the clmm function from the ordinal package. I conducted the following:

ordinal.GMSI <- clmm(response ~ condition * probe.position + (1|pp), data = ratingdata)

Now I would like to obtain some omnibus statistics for the main effects and interactions. I first did this manually, by defining models leaving one factor out and then comparing them, for example for the factor condition:

ordinal.minus.condition <- clmm(response ~ probe.position + (1|pp), data = ratingdata)
maineffectcondi <- anova(ordinalresults,ordinal.minus.condition)

This gives me a result showing a likelihood ratio test (chi squared):

Likelihood ratio tests of cumulative link models:

                    formula:                                         link: threshold:
ordinal.minus.condition response ~ probe.position + (1 | pp)             logit flexible  
ordinalresults          response ~ condition * probe.position + (1 | pp) logit flexible  

                    no.par   AIC logLik LR.stat df Pr(>Chisq)    
ordinal.minus.condition      9 41335 -20658                          
ordinalresults              21 38600 -19279  2758.9 12  < 2.2e-16 ***

However, I can also look at the main effects using the emmeans package and the joint_tests function. This seems much easier, especially since I may start adding factors to the model, and doing everything manually then quickly becomes a lot of work. I would then do this:


which gives me:

model term               df1 df2 F.ratio p.value
condition                  2 Inf 158.387 <.0001 
probe.position             5 Inf 236.343 <.0001 
condition:probe.position  10 Inf 231.791 <.0001

Here, the factor condition is again quite significant, but it's an F-test, not a Chi-square test. So which is the correct to use?

ps. I'm new to R and new to this forum, so please let me know if my question is unclear, inappropriate, or otherwise problematic!


A few points related to this question:

  1. All statistical tests on categorical data, including ordinal data, are asymptotic; that is, the stated distribution is only an approximation to the true distribution of the test statistic, and the approximation gets better as the sample size increases.

  2. There is a whole lexicon of chi-square tests. The one displayed in the anova() result is based on a likelihood ratio (the theory is that -2 times the log likelihood ratio is asymptotically chi-square). There are other chi-square tests out there. For contingency tables, for example, there are at least three in common use: the likelihood-ratio test, the Pearson chi-square (the most familiar on based on $\sum(O-E)^2/E$), and the Freeman-Tukey chi-square. All three have the same degrees of freedom, and get at the same thing, but have different values.

  3. An $F$ test with infinite denominator degrees of freedom is in fact a chi-square test, in the sense that if $F\sim F(\nu_1,\infty)$, then $\nu_1\cdot F \sim \chi^2(\nu_1)$. Thus, for the $F$ test of probe.position in the question, it is exactly the same as a chi-square test with the statistic $X^2 = 5\times236.343 \approx 1181.7$, with 5 d.f.

  4. The $F$ statistics computed by joint_tests() are more akin to Wald tests, which also have asymptotic chi-square distributions. See the Wikipedia article or some such.

  5. The anova() result shown is a comparison of a model with just probe.position as a fixed effect, and a model with fixed effects for probe.position, condition, and condition:probe.position. Thus it is not a test of the condition effect; it is a test of the combined effects of condition and condition:probe.position. Note that it has 12 degrees of freedom, which you can see is the sum of the degrees of freedom for these two terms in the joint_tests() table. Accordingly, we can cobble together a chi-square statistic for these two effects combined as $X^2 = 2\times158.387 + 10\times231.791 \approx 2634.7$, with $2+10=12$ d.f. This is actually in the same ballpark as the chi-square from the anova(); but the validity of this cobbled-together test depends on the condition and condition:probe.position effects being independent---which probably isn't exactly true.

In summary, there are many ways to get a chi-square test, and they will yield somewhat different results because they are computed differently and because all of them are approximate. In addition, we have to be careful to make sure we are testing the same thing. I hope this helps shed light on the question.

  • $\begingroup$ Yes, this is very helpful and answers my questions, thank you so much for taking the time to respond! $\endgroup$ – FBouwer Apr 4 '19 at 9:11
  • $\begingroup$ Very well explained, Russ. I am just wondering what method joint_tests() uses to get the denominator of df for pvals calculations? Given that is more of Wald test, is it possibly "inner-outer rule"? $\endgroup$ – Salahadin Lotfi May 25 '20 at 5:04
  • $\begingroup$ The denom df in the illustration are all Inf (infinite). These are asymptotic tests. Multiply those F ratios by their numerator df and you have chi-square tests. $\endgroup$ – Russ Lenth May 25 '20 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.