I'm a bit stumped as to what test(s) is/are most appropriate for the following scenario. I found a couple of similar questions but wasn't sure if the answers applied.

For a report I am working on for my boss, I am trying to test whether there is a statistically significant difference (and find the effect size) between the official letter grades earned by students who took a course during Spring semester taught using a traditional lecture model and the grades of students who took the same course during Spring semester following a redesign (i.e., determine if performance in the course is independent of course design). I also want to determine for which individual letter grades there is a significant difference (i.e., proportion of A's pre versus post-redesign, etc.). For the moment, I am just using 1 IV and 1 DV. (I've not completely wrapped my head around ordinal logistic regression models and diagnostics in Stata yet) :)

A bit more about the data:

  • Records in the data set represent individual students. I have the benefit of having access to the data for all students who have taken the course--not just a subset.
  • The DV (course grade) takes on the values: "A", "B", "C", "D", "F", and "W" (withdrawal), coded from -1 ("W) to 4 ("A"). The variable is really more ordinal than truly interval, since there is not exactly the same "distance" between all grades (from A to C != D to W, for instance).
  • The IV (course design) has two levels: before and after course redesign.
  • For Spring semester, N = 1717 students who took the traditional course and N = 44 students who took it post-redesign.
  • Spring grades are non-normally distributed, and the two groups have unequal variances.
  • a cross-tabulation of grades and course design for spring semester that includes expected counts shows 3 of 12 cells with an expected frequency < 5, 1 cell with expected frequency < 1, so I know I'd be violating two of the assumptions of a Chi^2 Test of Independence

I compared Fall:Fall using a Chi^2 test for independence, calculating Kendall's tau-b as a measure of effect size (not sure if gamma would have been better?), and looking at the standardized residuals in each column of the crosstab (using tabchi package) to determine significant difference for individual letter grades.

What are the most appropriate steps to take to analyze the Spring semester data?


As you recognise, grade is ordinal and not interval. It follows that variance is not well defined and tests of normality do not really apply. For example, the grading of "withdrawal" as $-1$, and indeed any other grading, is arbitrary other than respecting the order and such results are highly sensitive to the particular grading you use. Indeed, it's a widespread view that normality tests don't make much sense when applied to a 6-level ordinal variable.

Chi-square tests could be applied here, although in turn they won't pay any attention to the ordinal nature of the variable. The problem of small frequencies is real, but it is not so much an assumption as a condition that stretches the test somewhat; in any case a Fisher exact test is an alternative if that is a major concern.

I don't see any defensible alternative to tests that respect, or are consistent with, the ordinal nature of your response, and ordinal logit certainly seems a candidate. The possibility that your smaller group is too small to say much reliably about any differences is with you whatever you do.

There are many, many threads here on what to do with such data. Likert scale is a search term as well as ordinal scale.

  • $\begingroup$ I've redone the crosstab with the exact command to calculate significance for Fisher's exact test, thanks, and I will do some more reading about ordinal logit. I know gamma and tau-b do use the ordinal nature of variables; would they be inappropriate measures of association? I'm not clear on whether the number of levels of the IV matters but Acock (2012) cautions in A Gentle Introduction to Stata that the asymptotic standard errors reported by Stata "are only good estimates when you have a large sample" (p. 136). Thanks. $\endgroup$ – LASH Mar 18 '15 at 23:28
  • $\begingroup$ My understanding is that tau-b and gamma are possible measures of association for your case. Bootstrapping could be used to get at the uncertainty. $\endgroup$ – Nick Cox Mar 19 '15 at 1:02
  • $\begingroup$ Good to know! I've not used bootstrapping before, but from what I just read, it sounds very handy. I don't yet know how to go about applying it in this case--in his Gentle Intro to Stata text, Acock (2012) only discusses bootstrapping in relation to linear regression commands--so I've clearly more searching and reading to do. Is there a straightforward way to do this (or a particular resource I should check out?)? @MichaelChernick has a book available on Amazon I'll add to my wishlist, but it looks a bit advanced for the likes of me. :) Perhaps it's time to head over to Statalist. $\endgroup$ – LASH Mar 19 '15 at 2:15
  • $\begingroup$ Indeed. If you want specific advice on bootstrapping in Stata, ask on Statalist. If you want to read a book, find Efron and Tibshirani. $\endgroup$ – Nick Cox Mar 19 '15 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.