# Assessing differences in ratings between two groups

I'm trying to come up with an approach to determine if end-of-year performance ratings were given out fairly without bias. We have 3 ratings, below, meets, exceeds and these can be substituted to be just 1,2, or 3. The goal is to ensure that managers are not giving our more exceeds to males vs females. We don't have a force distribution so I can't compare the % of exceeds to some forced distribution. We'd like to do this per manager at the n-3 level, so 2 levels below the CEO. Typical sample sizes at this level can be a couple 100 to 20. Just depends on the department.

In years past, I've used a simple t.test to determine if the average rating for males is different than females. While this isn't ideal, it was the best I could come up with in a short time.

I know Kolmogorov-Smirnov is a possibility, but unsure of the assumptions needed to meet it's requirement and the interpretation of the output. My primary tools are R and Tableau for sharing the results.

• Are different managers rating the same persons?
– jkd
Nov 27, 2019 at 18:01
• Nope. One manager rating their own team. Nov 27, 2019 at 18:02
• One complication, of course, is that the results could look like a manager is acting with bias, whereas perhaps it is the case that employees of one gender, on that team, are simply performing better or worse. Maybe that's not a real concern if you have at least 20 employees per team. Nov 27, 2019 at 18:59
• t test is probably not appropriate for this kind of data. KS is probably not very useful here. I agree with @jkd that you may want a traditional nonparametric test. (Or ordinal regression if you want to sound fancy!) Nov 27, 2019 at 19:52

As I noted in my comments to the answer by @jkd , p values are affected by the sample size. With a large sample size, the test is likely to be sensitive to small differences between the two groups. This may treat managers with larger teams unfairly.

Therefore, it's important to assess differences between the groups by some kind of effect size statistic. This may be just comparing the proportions of each rating that each manager assigned to females and males. But also, standard effect size statistics can be helpful to capture and summarize the effect.

There are several options.

For both the r used for Wilcoxon-Mann-Whitney and Kendall's tau, a value of 0 indicates no effect, while values closer to 1 or -1 represent a large effect. Because of the setup of the groups here, a positive sign for r and a negative sign for tau indicate that females were rated higher.

Vargha and Delaney's A is useful because it has a natural interpretation. It is simply the probability that a value in one group will be larger than a value in the other group. A value of 0.50, therefore, indicates no effect, while values close to 0 to 1 indicate a large effect. Here, a value greater than 0.50 indicates that females were rated more highly.

Interpretation for any of these statistics will depend on the context and the assessment of the user.

In the following code, I use the example data by @jkd. I assess each manager separately. Note that the grammar for the different functions vary. Also note, as a caveat, that I am the author of the package with the functions I use for r, tau, and VDA.

### Packages and data

if(!require(rcompanion)){install.packages("rcompanion")}

set.seed(3)
df <- data.frame(Manager=rep(c("Tom Morris","Helga Lorry","Marry Clint"),times=c(12,20,5)),
Team.member.sex=sample(c("male","female"),37,replace=T),
Rating=ordered(sample(c("below","meets","exceeds"),37,replace=T),levels=c("below","meets","exceeds")))

df$$Rating.num = as.numeric(df$$Rating)
df$$Sex.num = as.numeric(df$$Team.member.sex)

### Effect sizes

df1=df[df$Manager=="Helga Lorry",] round(prop.table(xtabs(~ Team.member.sex + Rating, data=df1), margin=1), 2) ### Rating ### Team.member.sex below meets exceeds ### female 0.25 0.42 0.33 ### male 0.75 0.25 0.00 ### Note that each row sums to 1. library(rcompanion) wilcoxonR(df1$$Rating.num, df1$$Team.member.sex) ### r ### 0.521 ### Note that in many contexts, this is a pretty large value. ### Positive sign indicates females greater. spearmanRho(~ Rating.num + Sex.num, data=df1, method="kendall") ### tau ### -0.507 ### Note that in many contexts, this is a pretty large value. ### Negative sign indicates females greater. vda(Rating.num ~ Team.member.sex, data=df1) ### VDA ### 0.792 ### Note that in many contexts, this is a pretty large value (far from 0.50). ### Value > 0.50 sign indicates females greater. df2=df[df$Manager=="Marry Clint",]

round(prop.table(xtabs(~ Team.member.sex + Rating, data=df2), margin=1), 2)

###                Rating
### Team.member.sex below meets exceeds
###          female  0.25  0.50    0.25
###          male    0.00  1.00    0.00

wilcoxonR(df2$$Rating.num, df2$$Team.member.sex)

###  r
###  0

spearmanRho(~ Rating.num + Sex.num, data=df2, method="kendall")

###  tau
###    0

vda(Rating.num ~ Team.member.sex, data=df2)

### VDA
### 0.5

df3=df[df\$Manager=="Tom Morris",]

round(prop.table(xtabs(~ Team.member.sex + Rating, data=df3), margin=1), 2)

###                Rating
### Team.member.sex below meets exceeds
###          female  0.25  0.12    0.62
###          male    0.00  0.50    0.50

wilcoxonR(df3$$Rating.num, df3$$Team.member.sex)

###  r
###  0

spearmanRho(~ Rating.num + Sex.num, data=df3, method="kendall")

###  tau
###    0

vda(Rating.num ~ Team.member.sex, data=df3)

### VDA
### 0.5

• Thanks sal! The wilcoxon test seems like it’ll fit nicely because ideally we want to compare distribution to determine where the bias occurs. Ie are managers giving out more belows to females and more exceeds to males. The simple way I was going to do it was just compare the exceeds but this is a lot better. Nov 29, 2019 at 12:06
• Yes, but my point is to not read too much into a p value. For the same size effect, the test is more likely to find a significant result for a larger sample size than for a smaller sample size. This is easy to see if you take the same data, simply double it, and run the test again. This is why I said using a statistical test like this may be unfair to managers with larger teams. This is also why you need to look at effect size and practical importance. Nov 29, 2019 at 12:41
• There are other pitfalls as well. One thing is that if you are testing multiple managers, you may be likely to find a significant result just by chance. To buttress against this, you may want to use an adjustment on the p values to account for multiple tests. ... Also, because only one manager evaluates the team, you can never know if an imbalance in evaluations is the result of bias or just a fair assessment of imbalanced employees. ... Nov 29, 2019 at 13:19
• In any case, I would use caution in interpreting the results. The results may say less about individual managers and more about how the company overall is creating an environment where everyone is allowed to succeed. Nov 29, 2019 at 13:19
• Thanks for all of this! I was thinking of using VDA based on everything you said. I also saw that Mann-Whitney could be used here too. just so i understand the interpretation, am i correct in the following: if VDA = 0.792 (like your answer above) then the probability of a female is getting a higher rating is true here? Does this mean that females tended to get more exceeds or just higher ratings? Jan 6, 2020 at 18:34

You should rather use a non-parametric test here because the ratings are ordinal data.

I generated some example data:

set.seed(3)
df <- data.frame(Manager=rep(c("Tom Morris","Helga Lorry","Marry Clint"),times=c(12,20,5)),
Team.member.sex=sample(c("male","female"),37,replace=T),
Rating=ordered(sample(c("below","meets","exceeds"),37,replace=T),levels=c("below","meets","exceeds")))

sapply(levels(df$$Manager),function(x) { with(subset(df,Manager==x), kruskal.test(Rating~Team.member.sex))$$p.value
})
# Here it appears that Helga Lorry has some bias
# Helga Lorry Marry Clint  Tom Morris
#   0.0198458   1.0000000   1.0000000

library(ggplot2)
# this plot gives you the count of employees in each categorie (sex * rating)
# I prefered this to a stacked bar chart, because it is easier to see if a
# specific rating was never attributed to e.g. female employees
# but other kinds of visualization might be helpful as well
ggplot(df,aes(x=Team.member.sex,y=Rating)) +
geom_count(show.legend = F,) +
facet_wrap(.~Manager)


If you have a large number of managers you should probably adjust the p values in some way to avoid having too many false positives (e.g. using the p.adjust function). And of course you should always check the plots, do not consider the p values as absolute truth!

• One thing to keep in mind here is that the p value will be affected by the sample size. With a large sample size, the test is likely to be sensitive to small differences between the two groups. In a sense, this may treat managers with larger teams unfairly. Therefore, it's important to assess differences between the groups by some kind of effect size statistic. (And maybe give this more weight than the p value (!). Given an ordinal response across two groups, there are a variety of statistics that could be used. For statistics that may sound familiar for the audiance, Nov 27, 2019 at 19:55
• there is an r that is used for Wilcoxon-Mann-Whitney. There's also Kendall's tau-b (b because there are ties). A statistic that's less familiar but is actually more intuitive is Vargha and Delaney's A. It is simply the probability of an observation in one group being larger than an observation in the other group. (So 0.50 means the the two groups are stochastically equal.) Another simple way to report effect size may simply be the % of responses that are "exceeds" and "below" for each group. Nov 27, 2019 at 20:01
• @jkd Looks like your code doesn't actually work. When I run your code, I get an empty List() as a product Jan 14, 2020 at 17:58
• @TedMosby works for me... Did wou run the full code or try to adapt it to your data?
– jkd
Jan 14, 2020 at 18:25