3
$\begingroup$

I have letter grades (A through F) for two groups of students (Group 1 and 2) who both took the same classes at the same time. None of the students overlap in either of the groups. I have this data for about 11 separate courses. In some courses the number of students in Group 1 > Group 2 (1117 vs 126). How do I tell if the grade distributions between these groups are significantly different? (I want to see if the overall performance b/w the groups is similar enough to say there is no difference between them.)

I did a Kolmogorov-Smirnov two-sample test using ks.test {stats} and ks.boot {Matching}, but I became concerned that the data should be weighed for the different grades? Or does that not matter in K-S because it's just looking at the distributions coming from the same parent and weights make no sense? Also is K-S no good for categorical data?

I tried Welch's two-sample t-test using t.test {stats}. but again, I became concerned about weighted means, esp. for a t-test it seemed important to include? -- so then I ran wtd.t.test {weights} which seemed good for awhile, until I realized it was giving low p's in cases where the group sizes were different (1117 vs 126; 1915 vs. 1371 (still not sure if that is much different?)).

I'm now wondering if chi-squared is what I needed all along for the binned data...I ran chisq.test(cbind(x,y)) {stats} and it is giving me low p-values for groups that weren't sig. before...again, should I worry about including weights? Or should I trust chisq.test?

my data looks like:

x <- c(526,577,537,141,77,1,3,37,16) #Group1
y <- c(401,424,376,85,33,1,2,41,8) #Group2
weights <- c(4,3,2,1,0,0,0,0,0) #A-F, and others like W for withdraw, I incomplete, etc.
$\endgroup$
  • $\begingroup$ "I want to see if the overall performance b/w the groups is similar enough to say there is no difference between them." — Significance tests can't help you with that: they can find evidence that things are different, but not that they're the same or similar. $\endgroup$ – Kodiologist Aug 17 '17 at 18:57
  • $\begingroup$ I think it is implied what I meant; the null hypothesis is that there is no difference between the two groups. To reject the null is to support the claim that they are different. Your comment is not very helpful. After more thought I am going to use chi squared and group the grades into pass/fail to avoid any bins being too small. $\endgroup$ – redsharpie Aug 17 '17 at 19:27
  • $\begingroup$ "the null hypothesis is that there is no difference between the two groups. To reject the null is to support the claim that they are different." Quite true; what I'm saying is that if you fail to reject the null hypothesis, you haven't obtained any evidence that there is no difference. $\endgroup$ – Kodiologist Aug 17 '17 at 19:45
  • $\begingroup$ A chi-squared null hypothesis would be that there is no association between variables. If p is >= 0.05 then we accept H0 and say there is no association between student groups and grades -- the groups are similar in terms of how they perform in the course. To reject H0 would be that there is some association between group and the course outcome. If grade is associated to group, then it can be said that the groups are different in some way if they are producing different results? (Again, this is a lovely philosophical debate, but you haven't offered any advice on what test I should use...) $\endgroup$ – redsharpie Aug 17 '17 at 21:52
  • $\begingroup$ "If p is >= 0.05 then we accept H0" — Don't do that. If you fail to reject the null hypothesis, you haven't obtained any evidence that there is no difference. "If grade is associated to group, then it can be said that the groups are different in some way if they are producing different results?" — Yes. $\endgroup$ – Kodiologist Aug 17 '17 at 23:46
1
$\begingroup$

I think the chi-square approach is reasonable, but I wouldn't reduce the table to just pass and fail. The following example uses only grades A - F, and uses Cramer's v as a measure of association.

The results of prop.table suggests that the proportions are similar across groups.

 if(!require(vcd)){install.packages("vcd")}

 x <- c(526,577,537,141,77,1,3,37,16) #Group1
 y <- c(401,424,376, 85,33,1,2,41, 8) #Group2

 x = x[1:5]
 y = y[1:5]

 Matrix           = as.matrix(rbind(x,y))
 rownames(Matrix) = c("Group.1", "Group.2")
 colnames(Matrix) = c("A", "B", "C", "D", "F")

 Matrix

 prop.table(Matrix, margin=1)

 chisq.test(Matrix)

 library(vcd)

 assocstats(Matrix)

But if you look at the results of prop.table again, it's clear that Group 2 has higher proportions in the better grades and lower in the proportions in the worse grades.

So, it is probably important to take into account the ordinal nature of the letter grades. This can be assessed with the Cochran-Armitage test. Freeman's theta or epsilon squared can be used as a measure of association.

 if(!require(coin)){install.packages("coin")}
 if(!require(rcompanion)){install.packages("rcompanion")}

 library(coin)

 Table = as.table(Matrix)

 names(dimnames(Table)) = c("Group", "Grade")

 Table

 chisq_test(Table,
           scores = list("Grade" = c(-2, -1, 0, 1, 2)))

 library(rcompanion)

 freemanTheta(Table)

 epsilonSquared(Table)
$\endgroup$
  • $\begingroup$ Thanks Sal! This detailed answer really helped me understand and learn! (P.S. I'm thrilled to see you work at my alma mater! I miss it!) $\endgroup$ – redsharpie Aug 18 '17 at 20:46
0
$\begingroup$

As we discussed in the comments, if you want to show that the groups are similar, significance testing won't help. So, what then? Confidence intervals (CIs) will be more useful. You can look at a CI for the effect of group and judge for yourself whether the effect is small enough that the groups have sufficiently similar performance for your purposes.

So how do you get this CI? You could set up a linear model such that the CI for group is the CI you want. Since you have two disjoint groups of students, and all the students take the same 11 courses, you could use a mixed model with each student and course having a random intercept, and group having a single (fixed-effect) dummy variable.

A difficulty is that the dependent variable (DV), letter grade, is on an unusual scale. Approaches to this include ordinal regression, plain linear regression with a transformed DV, and plain linear regression without a transformation and with the hope that your inferences are still sufficiently accurate. Use the sample size, a plot of the DV, and possibly plots of the residuals to make this decision.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.