Bias in jury selection?

Question

A friend is representing a client on appeal, after a criminal trial in which it appears that jury selection was racially biased.

The jury pool consisted of 30 people, in 4 racial groups. The prosecution used peremptory challenges to eliminate 10 of these people from the pool. The number of people and number of actual challenges in each racial group were, respectively:

A: 10, 1
B: 10, 4
C:  6, 4
D:  4, 1
total: 30 in pool, 10 challenges

The defendant was from racial group C and the victims from racial groups A and D, so the concern a priori is whether group C is over-challenged and groups A and D under-challenged. Legally (IIUC; IANAL), the defense does not need to prove racial bias, but merely to show that the data seem to indicate bias, which then puts the burden on the prosecution to explain each challenge non-racially.

Is the following analysis correct in its approach? (I think the calculations are fine.):

There are nCr(30,10) = 30,045,015 distinct sets of 10 pool members. Of these distinct sets, I count that 433,377 sets include both (no more than 2 members of group A and D combined) and (no fewer than 4 members of group C).

Thus the chance of reaching the observed level of apparent bias favoring groups A and D over group C (where favoring means not including in the set of 10 challenges) would be the ratio of these, 433/30045 = 1.44%.

Thus the null hypothesis (no such bias) is rejected at the 5% significance level.

If this analysis is methodologically correct, what would be the most succinct way to describe it to a court, including an academic / professional reference (i.e. not Wikipedia)? While the argument seems simple, how can one most clearly and succinctly demonstrate to the court that it's correct, not shenanigans?

---

Update: This question was under consideration as a tertiary argument in an appeal brief. Given the technical complexity (from the lawyer's viewpoint) of the discussion here and the apparent lack of legal precedent, the lawyer has chosen not to raise it, so at this point the question is mostly theoretical / educational.

To answer one detail: I believe that the number of challenges, 10, was set in advance.

After studying the thoughtful and challenging answers and comments (thanks, all!), it seems that there are 4 separate issues here. For me, at least, it would be most helpful to consider them separately (or to hear arguments why they are not separable.)

1) Is the consideration of the races of both defendant and victims, in the jury pool challenges, of legal concern a priori? The goal of the appeal argument would merely be to raise reasonable concern, which could lead to a judicial order that the prosecution state the reason for each individual challenge. This does not appear to me to be a statistical question, but rather a social / legal one, which is at the lawyer's discretion to raise or not.

2) Assuming (1), is my choice of an alternative hypothesis (qualitatively: bias against jurors who share the defendant's race, in favor of those who share the victims' races) plausible, or is it impermissibly post hoc? From my lay perspective, this is the most perplexing question -- yes, of course one would not raise it if one did not observe it! The problem, as I understand, is selection bias: one's tests should consider not just this jury pool but the universe of all such jury pools, including all the ones where the defense did not observe a discrepancy and therefore were not tempted to raise the issue. How does one address this? (For example, how does Andy's test address this?) It appears, though I may be wrong about this, that most respondents are not troubled by potentially post-hoc 1-tailed tests for bias solely against the defendant's group. How would it be methodologically different to simultaneously test bias for victim groups, assuming (1)?

3) If one stipulates my choice of a qualitative alternative hypothesis as stated in (2), then what is an appropriate statistic for testing it? This is where I am most puzzled by the responses, because the ratio that I propose seems to be a slightly more conservative analog of Andy's test for the simpler "bias against C" alternative hypothesis (more conservative because my test also counts all cases further out in the tail, not just the exact observed count.)

Both tests are simple counting tests, with the same denominator (same universe of samples), and with numerators corresponding precisely to the frequency of those samples which correspond to the respective alternative hypotheses. So @whuber, why is it not identically as true of my counting test as of Andy's that it "can be based on stipulated null [same] and alternative [as described] hypotheses and justified using the Neyman-Pearson lemma"?

4) If one stipulates (2) and (3), are there references in case law which would convince a skeptical appeals court? From the evidence to date, probably not. Also, at this stage of appeal there's no opportunity for any "expert witness", so references are everything.

Answer 1

Here's how I might approach answering your question using standard statistical tools.

Below are the results of a probit analysis on the probability of being rejected given the juror's group membership.

First, here's what the data looks like. I have 30 observations of group and a binary rejected indicator:

. tab group rejected 

           |       rejected
     group |         0          1 |     Total
-----------+----------------------+----------
         A |         9          1 |        10 
         B |         6          4 |        10 
         C |         2          4 |         6 
         D |         3          1 |         4 
-----------+----------------------+----------
     Total |        20         10 |        30 

Here are the individual marginal effects as well as the joint test:

. qui probit rejected ib2.group

. margins rb2.group

Contrasts of adjusted predictions
Model VCE    : OIM

Expression   : Pr(rejected), predict()

------------------------------------------------
             |         df        chi2     P>chi2
-------------+----------------------------------
       group |
   (A vs B)  |          1        2.73     0.0986
   (C vs B)  |          1        1.17     0.2804
   (D vs B)  |          1        0.32     0.5731
      Joint  |          3        8.12     0.0436
------------------------------------------------

--------------------------------------------------------------
             |            Delta-method
             |   Contrast   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
       group |
   (A vs B)  |        -.3    .181659     -.6560451    .0560451
   (C vs B)  |   .2666667   .2470567     -.2175557     .750889
   (D vs B)  |       -.15   .2662236     -.6717886    .3717886
--------------------------------------------------------------

Here we are testing the individual hypotheses that the differences in the probability of being rejected for groups A, C, and D compared to group B are zero. If everyone was as likely to be rejected as group B, these would be zero. The last piece of output tells us that group A and D jurors are less likely to be rejected, while group C jurors are more likely to be turned away. These differences are not statistically significant individually, though the signs agree with your bias conjecture.

However, we can reject the joint hypothesis that the three differences are all zero at $p=0.0436$.

---

Addendum:

If I combine groups A and D into one since they share the victims' races, the probit results get stronger and have a nice symmetry:

Contrasts of adjusted predictions
Model VCE    : OIM

Expression   : Pr(rejected), predict()

------------------------------------------------
             |         df        chi2     P>chi2
-------------+----------------------------------
      group2 |
 (A+D vs B)  |          1        2.02     0.1553
   (C vs B)  |          1        1.17     0.2804
      Joint  |          2        6.79     0.0336
------------------------------------------------

--------------------------------------------------------------
             |            Delta-method
             |   Contrast   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
      group2 |
 (A+D vs B)  |  -.2571429   .1809595      -.611817    .0975313
   (C vs B)  |   .2666667   .2470568     -.2175557     .750889
--------------------------------------------------------------

This also allows Fisher's exact to give congruent results (though still not at 5%):

 RECODE of |       rejected
     group |         0          1 |     Total
-----------+----------------------+----------
       A+D |        12          2 |        14 
         B |         6          4 |        10 
         C |         2          4 |         6 
-----------+----------------------+----------
     Total |        20         10 |        30 

          Pearson chi2(2) =   5.4857   Pr = 0.064
           Fisher's exact =                 0.060

Answer 2

I would think that introducing an ad hoc statistical method is going to be a no-go with the court. It is better to use methods that are "standard practice". Otherwise, you probably get to prove your qualifications to develop new methods.

To be more explicit, I do not think that your method would meet the Daubert standard. I also very much doubt that your method has any academic reference in and of itself. You would probably have to go the route of hiring a statistical expert witness to introduce it. It would be easily countered, I would think.

The basic question here is likely: "Was jury challenge independent of racial grouping?"

These are small numbers to which to be applying asymptotically based statistical methods. However, the "standard" for testing association in this setting is the $\chi^2$ test:

> M <- as.table(cbind(c(9, 6, 2, 3), c(1, 4, 4, 1)))
> dimnames(M) <- list(Group=c("A", "B", "C", "D"), Challenged=c("No", "Yes"))
> M
     Challenged
Group No Yes
    A  9   1
    B  6   4
    C  2   4
    D  3   1

> chisq.test(M)

        Pearson's Chi-squared test

data:  M
X-squared = 5.775, df = 3, p-value = 0.1231

Warning message:
In chisq.test(M) : Chi-squared approximation may be incorrect

Using the Fisher exact test gives similar results:

> fisher.test(M)

        Fisher's Exact Test for Count Data

data:  M
p-value = 0.1167
alternative hypothesis: two.sided

The note about the hypothesis being two-sided applies to the case of a $2\times2$ table.

My interpretation is that there is not much evidence to argue racial bias.

Answer 3

I asked a similar question previously (for reference here is the particular case I discuss). The defense needs to simply show a prima facia case of discrimination in Batson challenges (assuming US criminal law) - so hypothesis tests are probably a larger burden than is needed.

So for:

• $n = 30$ people on the venire panel
• $p = 6$ people of racial class C on the panel
• $k = 4$ jurors of racial group C eliminated on preemptory challenges
• $d = 10$ preemptory challenges

Whuber's previous answer gives the probability of this particular outcome being dictated by the hypergeometric distribution:

$$\frac{{p \choose k} {n-p \choose d-k} }{{n \choose d}}$$

Which Wolfram-Alpha says equals in this case:

$$\frac{{6 \choose 4} {30-6 \choose 10-4} }{{30 \choose 10}} = \frac{76}{1131} \approx 0.07$$

Unfortunately I do not have a reference besides the links I have provided - I imagine you can dig up a suitable reference for the hypergeometric distribution from the Wikipedia page.

This ignores the question about whether racial groups A and D are "under-challenged". I'm skeptical you could make a legal argument for this -- it would be a weird twist on the equal protection clause, This particular group is too protected!, that I do not think would fly. (I'm not a lawyer though - so take with a grain of salt.)

If you really want a hypothesis test I am not sure how to go about it. You can generate the $30 \choose 10$ permutations, give it a probability under the null of racial groups being equally chosen per their proportions in the venire, and then calculate the exact distribution of your test statistic under the null. I'm not quite sure what test statistic is satisfactory though, $\chi^2$ doesn't really answer the question of interest. (Is it alright you make up your own test statistic -- I do not know?)

---

I've updated some of my thoughts in a blog post. My post is specific to Batson Challenges, so it is unclear if you seek another situation (your updates for 1 and 2 don't make sense in the context of Batson Challenges.)

I was able to find one related article (available in full at the link):

Gastwirth, J. L. (2005). Case comment: statistical tests for the analysis of data on peremptory challenges: clarifying the standard of proof needed to establish a prima facie case of discrimination in Johnson v. California. Law, Probability and Risk, 4(3), 179-185.

That gave the same suggestion for using the hypergeometric distribution. In my blog post I show how if you collapse the categories into two groups it is equivalent to Fisher's Exact test.

Gastwirth suggests as I did in my comment that you could consider $k$ as the test statistic, and so add the probability of $k = 5$ and $k = 6$ to that above if you prefer. Gastwirth also gives an example for calculating a test statistic based on changing numbers of $n$ in the jury pool. In my blog post I just conduct sensitivity analysis for varying levels of $n$ and $d$ (for a different case) to provide ranges of possible percentages.

If someone becomes aware of case law that actually uses this (or anything besides fractions) I would be interested.

Answer 4

Let's not forget the multiple testing issue. Imagine 100 defense lawyers each looking for grounds to appeal. All of the juror rejections had been performed by flipping coins or rolling dice for each prospective juror. Therefore, none of the rejections were racially biased.

Each of the 100 lawyers now does whatever statistical test all of you guys agree on. Roughly five out of that 100 will reject the null hypothesis of "unbiased" and have grounds for appeal.