What test(s) should I use to compare 2, non-normally distributed populations with unequal variances? (IV is 2 categories; DV is ordinal)

Question

I'm a bit stumped as to what test(s) is/are most appropriate for the following scenario. I found a couple of similar questions but wasn't sure if the answers applied.

For a report I am working on for my boss, I am trying to test whether there is a statistically significant difference (and find the effect size) between the official letter grades earned by students who took a course during Spring semester taught using a traditional lecture model and the grades of students who took the same course during Spring semester following a redesign (i.e., determine if performance in the course is independent of course design). I also want to determine for which individual letter grades there is a significant difference (i.e., proportion of A's pre versus post-redesign, etc.). For the moment, I am just using 1 IV and 1 DV. (I've not completely wrapped my head around ordinal logistic regression models and diagnostics in Stata yet) :)

A bit more about the data:

• Records in the data set represent individual students. I have the benefit of having access to the data for all students who have taken the course--not just a subset.
• The DV (course grade) takes on the values: "A", "B", "C", "D", "F", and "W" (withdrawal), coded from -1 ("W) to 4 ("A"). The variable is really more ordinal than truly interval, since there is not exactly the same "distance" between all grades (from A to C != D to W, for instance).
• The IV (course design) has two levels: before and after course redesign.
• For Spring semester, N = 1717 students who took the traditional course and N = 44 students who took it post-redesign.
• Spring grades are non-normally distributed, and the two groups have unequal variances.
• a cross-tabulation of grades and course design for spring semester that includes expected counts shows 3 of 12 cells with an expected frequency < 5, 1 cell with expected frequency < 1, so I know I'd be violating two of the assumptions of a Chi^2 Test of Independence

I compared Fall:Fall using a Chi^2 test for independence, calculating Kendall's tau-b as a measure of effect size (not sure if gamma would have been better?), and looking at the standardized residuals in each column of the crosstab (using tabchi package) to determine significant difference for individual letter grades.

What are the most appropriate steps to take to analyze the Spring semester data?

Answer 1

As you recognise, grade is ordinal and not interval. It follows that variance is not well defined and tests of normality do not really apply. For example, the grading of "withdrawal" as $-1$, and indeed any other grading, is arbitrary other than respecting the order and such results are highly sensitive to the particular grading you use. Indeed, it's a widespread view that normality tests don't make much sense when applied to a 6-level ordinal variable.

Chi-square tests could be applied here, although in turn they won't pay any attention to the ordinal nature of the variable. The problem of small frequencies is real, but it is not so much an assumption as a condition that stretches the test somewhat; in any case a Fisher exact test is an alternative if that is a major concern.

I don't see any defensible alternative to tests that respect, or are consistent with, the ordinal nature of your response, and ordinal logit certainly seems a candidate. The possibility that your smaller group is too small to say much reliably about any differences is with you whatever you do.

There are many, many threads here on what to do with such data. Likert scale is a search term as well as ordinal scale.