2
$\begingroup$

I am conducting an analysis to see whether stop and searches in the UK have bias towards minorities.

The data looks like the following:

ethnicity    actionTaken
Black         Action taken
White         Action taken
Asian         No action taken
Other         No action taken  

There are 200k entries in my data and there are also other metrics like age, gender, location, etc. The action taken variable has 14 different levels, like: cautioned,arrested,court summoned, no action taken, so I divided these into two levels as shown above: action taken and no action taken. The reason I did this is to create a logistic regression model.

My (first) method is to determine whether there exists bias via "hit rates" (which is taken from this paper). Hit rate is calculated by the amount of successful stop and searches divided by the total amount of hit rates. Lower hit rates means lower standards of suspicion are applied when deciding to conduct a stop and search procedure. The first thing I do is I calculate the hit rates of each ethnicity and then via Chi-squared tests for Homogeneity I determine whether there are significant differences in the hit rates.

When it comes to Others (ethnicities), I have the following table:

table = as.table(rbind(c(1925,476),c(721,233)))
names(dimnames(table)) = c("Ethnicity","Outcome")
rownames(table) = c("White","Other")
colnames(table) = c("Successful", "Unsuccessful")
chisq.test(table)

Which produces the following output:

Pearson's Chi-squared test with Yates' continuity correction

data:  table
X-squared = 8.3882, df = 1, p-value = 0.003777

So, this means that there is a significant difference in proportion (i.e. hit rates), so this means that there is a tendency towards bias for police officers to apply lower standards of suspicion to Others, compared to White. (Just for reference, the hit rate for Whites is 80% and for others it is 75%).

Back to the logistic regression: I create a model and I get the following output:

Call:
glm(formula = OutcomeClassFac ~ ethnicity, family = "binomial", 
    data = df[df$Force == "nottinghamshire", ])

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.9660   0.5593   0.6648   0.7319   0.7484  

Coefficients:
               Estimate Std. Error z value Pr(>|z|)    
(Intercept)     1.39726    0.05119  27.296   <2e-16 ***
ethnicityAsian  0.37887    0.17266   2.194   0.0282 *  
ethnicityBlack -0.21693    0.11028  -1.967   0.0492 *  
ethnicityOther -0.26766    0.09110  -2.938   0.0033 ** 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 4352.3  on 4234  degrees of freedom
Residual deviance: 4333.3  on 4231  degrees of freedom
AIC: 4341.3

Number of Fisher Scoring iterations: 4

And I explain the coefficients in the following way:

  1. The odds of having a successful stop and search for the police on Asian people over the odds of having a successful stop and search for the police on White people is $\exp(-0.378)=1.46$. In terms of percent change, we can say that the odds for Asians are 46% higher than the odds for Whites, clearly indicating that there are no lower standards of suspicion applied to Asians.
  2. Following the same method as above the odds for Black people over White people are $\exp(-0.216)=0.80$, i.e 20% lower than the odds for Whites, which could indicate bias.
  3. For Other people the odds over White people are $\exp(0.267)=.76$, i.e 24% lower than the odds for Whites, which could indicate bias.

I then do hypothesis testing to see whether Blacks and Others are indeed getting disparate treatment:

# Testing whether the coefficients of White and Blacks are significantly different. We will not set the Whites as a reference so that we can use their coefficient when conducting the hypothesis
df$ethnicity <- factor(df$ethnicity)
# We create a model with no intercept
mylogitNot <- glm(OutcomeClassFac ~  0 + ethnicity, data = df[df$Force == "nottinghamshire",], family = "binomial")
# We will now create a vector "l" that defines the test we want to perform. In this case, we want to test the difference (subtraction) of the terms for White and Black ethnicities (i.e., the 1st and 3rd terms in the model). To contrast these two terms, we multiply one of them by 1 and the other by -1. The other terms are not involved in the test, so they are multiplied by 0. 
l <- cbind(1, 0, -1, 0)
# We now conduct the hypothesis test
library(aod)
wald.test(b = coef(mylogitNot), Sigma = vcov(mylogitNot), L = l)
# Test for Others
l <- cbind(1, 0, 0, -1)
wald.test(b = coef(mylogitNot), Sigma = vcov(mylogitNot), L = l)

The test for Others returns:

Wald test:
----------

Chi-squared test:
X2 = 8.6, df = 1, P(> X2) = 0.0033

So this indicates that the coefficients are statistically significant and hence there's Bias towards Others. Hence, this modelling coincides with the results I get from the method I read in the paper linked above.

Questions:

Would this be an appropriate method?

Am I doing anything wrong with the modelling part?

$\endgroup$
4
  • 1
    $\begingroup$ With 200k obs you should not throw together all those categories, keep them. You can still use (multinomial) logistic regression. $\endgroup$ Commented Mar 5, 2021 at 14:24
  • $\begingroup$ @kjetilbhalvorsen Wouldn't that be too hard to interpret, with 14 levels? Out of the 14 levels, 3-4 are equivalent to no action taken and the rest are action taken. So, how would it be possible to interpret such results with mixed outcomes? Also, since we can calculate the proportion of successful to overall searches when they are in a bivariate form, how would it be possible to do this calculation for separate outcomes? For example, how would I calculate the hit rate for cautioned or arrested? $\endgroup$
    – user274779
    Commented Mar 5, 2021 at 15:59
  • $\begingroup$ Because you used ~0+, the coefficients aren't testing the rates for the ethnicities against white, they are testing the rates for the ethnicities (including white) against 50%. $\endgroup$ Commented Mar 5, 2021 at 17:51
  • $\begingroup$ @gung-ReinstateMonica I included the wrong output. Now it's the correct output (the summary output) $\endgroup$
    – user274779
    Commented Mar 5, 2021 at 18:00

1 Answer 1

0
$\begingroup$

I am not quite sure whether I understood all details correctly but regarding your logistic regression:

From the description I assume that the response variable in your model ("OutcomeClassFac") stands for a factor with the two levels "successful stop" and "unsuccessful stop". Since you fit a logistic regression with no intercept, the 4-level factor "ethnicity" is represented in the model with four dummy variables, one for each ethnicity. Therefore, each ethnicity gets its own coefficient, which represents the log-odds of the associated ethnic group. As you correctly state, the estimated Asian log-odds are for example higher than the Whites log-odds which would indicate that there is a higher probability of a successful stop in Asians. You then proceed to investigate whether these differences are statistically significant by using Wald-tests. In general I think this procedure is okay.

Still, if you are mainly interested in the differences between the groups, then I think it would be easier to fit a logistic regression including an intercept. The reference level, in your case "White", would be represented by the intercept. The three coefficients "Asian", "Black" and "Other" would describe the differences in log-odds when moving from "Whites" to the respective group. Like this you get the differences (with regard to the reference level) and their statistical significance in one go. Also, this would easily allow you to include potential confounders in your model in order to "adjust" the differences between ethnic groups by controlling for these confounder variables. Perhaps I'm wrong but it sounds a bit like you want to give the shown effects somewhat of a causal meaning. Even though this is a very difficult endeavor with observational data in general, one should at least try to statistically control for potential confounders when trying to suggest causal links.

$\endgroup$
5
  • $\begingroup$ I don't like this answer. Of course confounding factors are good, but using a reference class to make hypothesis tests on the other effects is almost never a good idea. It means you are testing the possibility that two groups are actually one in the data. That doesn't make any real sense in most situations, and is conceptually better done trough a test with contrasts anyway. $\endgroup$
    – carlo
    Commented Mar 13, 2021 at 17:13
  • $\begingroup$ I can't quite follow your argumentation. The two approaches are conceptually the same thing, no? In one approach one estimates the log-odds of both groups and tests whether they are equal (contrast). In the other one tests whether their difference is equal to zero (coefficient). In the posted question one can compare the p-value of the coeffient of "ethnicityOther" and the p-value of the contrast for "ethnicityOther" (vs. Whites). They are the same value. So I don't see how one is better or worse than the other. $\endgroup$
    – YR2018
    Commented Mar 15, 2021 at 20:58
  • $\begingroup$ then why are you suggesting one over the other? I'm suggesting the other one because it's easy to forget that there is a baseline category. Also not fitting the intercept reflects the actual data in a more straightforward way, and does not invite you to make this kind of tests, like the OP did with contrasts, that don't reflect any sensible scenario (why would blacks and white have the same stats, while asians and others no?). $\endgroup$
    – carlo
    Commented Mar 16, 2021 at 16:09
  • $\begingroup$ As clearly stated in my answer, I suggested it because it seems "easier" to me. Easier because it returns all the comparisons the researcher seemed to be interested in with one command. Regarding your point that such tests "don't reflect any sensible scenario" I am not quite sure what you mean. You mean that its unlikely that the data follows a structure dictated by a linear model? $\endgroup$
    – YR2018
    Commented Mar 19, 2021 at 18:07
  • $\begingroup$ no, I mean that it's unlikely that black people and white people have the same population proportion, while asians and others are allowed to have different parameters. I guess a unilateral test may be presented better, but this way the test IMO reflects no sensible decision scenario. $\endgroup$
    – carlo
    Commented Mar 21, 2021 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.