Are two grade distributions significantly different in R

Question

I have letter grades (A through F) for two groups of students (Group 1 and 2) who both took the same classes at the same time. None of the students overlap in either of the groups. I have this data for about 11 separate courses. In some courses the number of students in Group 1 > Group 2 (1117 vs 126). How do I tell if the grade distributions between these groups are significantly different? (I want to see if the overall performance b/w the groups is similar enough to say there is no difference between them.)

I did a Kolmogorov-Smirnov two-sample test using ks.test {stats} and ks.boot {Matching}, but I became concerned that the data should be weighed for the different grades? Or does that not matter in K-S because it's just looking at the distributions coming from the same parent and weights make no sense? Also is K-S no good for categorical data?

I tried Welch's two-sample t-test using t.test {stats}. but again, I became concerned about weighted means, esp. for a t-test it seemed important to include? -- so then I ran wtd.t.test {weights} which seemed good for awhile, until I realized it was giving low p's in cases where the group sizes were different (1117 vs 126; 1915 vs. 1371 (still not sure if that is much different?)).

I'm now wondering if chi-squared is what I needed all along for the binned data...I ran chisq.test(cbind(x,y)) {stats} and it is giving me low p-values for groups that weren't sig. before...again, should I worry about including weights? Or should I trust chisq.test?

my data looks like:

x <- c(526,577,537,141,77,1,3,37,16) #Group1
y <- c(401,424,376,85,33,1,2,41,8) #Group2
weights <- c(4,3,2,1,0,0,0,0,0) #A-F, and others like W for withdraw, I incomplete, etc.

Answer 1

I think the chi-square approach is reasonable, but I wouldn't reduce the table to just pass and fail. The following example uses only grades A - F, and uses Cramer's v as a measure of association.

The results of prop.table suggests that the proportions are similar across groups.

 if(!require(vcd)){install.packages("vcd")}

 x <- c(526,577,537,141,77,1,3,37,16) #Group1
 y <- c(401,424,376, 85,33,1,2,41, 8) #Group2

 x = x[1:5]
 y = y[1:5]

 Matrix           = as.matrix(rbind(x,y))
 rownames(Matrix) = c("Group.1", "Group.2")
 colnames(Matrix) = c("A", "B", "C", "D", "F")

 Matrix

 prop.table(Matrix, margin=1)

 chisq.test(Matrix)

 library(vcd)

 assocstats(Matrix)

But if you look at the results of prop.table again, it's clear that Group 2 has higher proportions in the better grades and lower in the proportions in the worse grades.

So, it is probably important to take into account the ordinal nature of the letter grades. This can be assessed with the Cochran-Armitage test. Freeman's theta or epsilon squared can be used as a measure of association.

 if(!require(coin)){install.packages("coin")}
 if(!require(rcompanion)){install.packages("rcompanion")}

 library(coin)

 Table = as.table(Matrix)

 names(dimnames(Table)) = c("Group", "Grade")

 Table

 chisq_test(Table,
           scores = list("Grade" = c(-2, -1, 0, 1, 2)))

 library(rcompanion)

 freemanTheta(Table)

 epsilonSquared(Table)

Answer 2

As we discussed in the comments, if you want to show that the groups are similar, significance testing won't help. So, what then? Confidence intervals (CIs) will be more useful. You can look at a CI for the effect of group and judge for yourself whether the effect is small enough that the groups have sufficiently similar performance for your purposes.

So how do you get this CI? You could set up a linear model such that the CI for group is the CI you want. Since you have two disjoint groups of students, and all the students take the same 11 courses, you could use a mixed model with each student and course having a random intercept, and group having a single (fixed-effect) dummy variable.

A difficulty is that the dependent variable (DV), letter grade, is on an unusual scale. Approaches to this include ordinal regression, plain linear regression with a transformed DV, and plain linear regression without a transformation and with the hope that your inferences are still sufficiently accurate. Use the sample size, a plot of the DV, and possibly plots of the residuals to make this decision.