1
$\begingroup$

I've been handed a bunch of historical data on performance ratings of individuals in a number of organizations, and asked to identify any instances where there appear to be "abnormalities" in the distribution of ratings. This is not to immediately throw a "bias flag" but to identify cases where more scrutiny may be called for.

For privacy reasons the data below is notional, but representative of one of the orgs in question. Many of them are significantly larger than this, but I chose to represent one of the smaller ones thinking that a methodology would "scale up" better than down...

At this location, the racial breakdown of the ratings is

Race Count Percent
Black 47 5.0%
Hispanic 18 1.9%
Other 24 2.5%
White 855 90.6%

Individuals are given a rating of A, B, or C. In the aggregate those broke down as

Rating Count Percent
A 363 38.5%
B 482 51.1%
C 2 0.2%

And finally, by racial category

Rating Black Hispanic Other White
A 17 4 7 335
B 24 11 15 432
C 0 1 0 1

My initial thought was to look first at the population proportions...so for instance black people were 5% of the population, so you should expect them to be roughly %5 of each rating. And for this data they were 4.7% of the A's, 5% of the B's, and 0% of the C's but there were only 2 C's. So for A's there's a discrepancy of 0.3% from expected and 0% for B's.

It was then put to me that I should start instead with the distribution of the ratings. Since 38.5% of the ratings given were A's, roughly 38.5% of black ratings should have been an A. And in this case 41.6% of black ratings were A's. That's a discrepancy of 3.1% which, although not huge, is certainly more than the 0.3% from the other method.

And in either case, I have no idea how to talk about statistical significance.

So I'm hoping someone can point me to a resource or outline an appropriate methodology to make these comparisons in a defensible manner.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I think the best way to approach this would just be to use the last contingency table you provide and to utilize Pearson's Chi-squared test (or a similar test). Here, you're testing whether the rows in your column are independent of the columns, i.e. do these racial categories tell us anything about the rankings.

This is how you'd do it in R:

tab <- matrix(data = c(17, 24, 0, 4, 11, 1, 7, 15, 0, 335, 432, 1),
              nrow = 3,
              ncol = 4)

chisq.test(tab, simulate.p.value = TRUE)

Simulating p-values to avoid relying on the asymptotic approximation to the Chi-squared distribution, we get a p-value of 0.021.

Improve this answer
$\endgroup$
7
  • $\begingroup$ If I understand correctly, that result would lead me to conclude that they are NOT independent and therefore the racial categories are "significant" in determining the rates, but doesn't allow me to say anything more specific with regard to individual categories, correct? So I can't really comment on whether, say, black people are awarded A's at a higher/lower rate. $\endgroup$
    – jerH
    Commented Feb 23, 2022 at 4:50
  • $\begingroup$ Correct but for this you can look at the proportions as you suggest above. The difference between your two methods is just the difference in whether you're looking at the joint probabilities (dividing by the total count) or the marginal probability (dividing by the total count for a particular column). $\endgroup$
    – num_39
    Commented Feb 23, 2022 at 5:05
  • $\begingroup$ So is there any approach that would allow conclusions such as "the other category is under/over - represented in A's at a statistically significant rate"? because ultimately I think that's what the "higher ups" are asking me for.... $\endgroup$
    – jerH
    Commented Feb 23, 2022 at 5:27
  • $\begingroup$ First, I should clarify that just looking at the proportions can be misleading b/c proportions based on smaller counts are less precise than proportions based on larger counts. So for a particular table, the largest difference in proportions isn't necessarily what's driving significance. $\endgroup$
    – num_39
    Commented Feb 23, 2022 at 5:41
  • $\begingroup$ Off the top of my head, I can't think of a good way to do this without running into problems with multiple comparisons, i.e. you start running multiple tests and your probability of false positives begins to increase. $\endgroup$
    – num_39
    Commented Feb 23, 2022 at 5:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.