# Bias in jury selection?

A friend is representing a client on appeal, after a criminal trial in which it appears that jury selection was racially biased.

The jury pool consisted of 30 people, in 4 racial groups. The prosecution used peremptory challenges to eliminate 10 of these people from the pool. The number of people and number of actual challenges in each racial group were, respectively:

A: 10, 1
B: 10, 4
C:  6, 4
D:  4, 1
total: 30 in pool, 10 challenges


The defendant was from racial group C and the victims from racial groups A and D, so the concern a priori is whether group C is over-challenged and groups A and D under-challenged. Legally (IIUC; IANAL), the defense does not need to prove racial bias, but merely to show that the data seem to indicate bias, which then puts the burden on the prosecution to explain each challenge non-racially.

Is the following analysis correct in its approach? (I think the calculations are fine.):

There are nCr(30,10) = 30,045,015 distinct sets of 10 pool members. Of these distinct sets, I count that 433,377 sets include both (no more than 2 members of group A and D combined) and (no fewer than 4 members of group C).

Thus the chance of reaching the observed level of apparent bias favoring groups A and D over group C (where favoring means not including in the set of 10 challenges) would be the ratio of these, 433/30045 = 1.44%.

Thus the null hypothesis (no such bias) is rejected at the 5% significance level.

If this analysis is methodologically correct, what would be the most succinct way to describe it to a court, including an academic / professional reference (i.e. not Wikipedia)? While the argument seems simple, how can one most clearly and succinctly demonstrate to the court that it's correct, not shenanigans?

Update: This question was under consideration as a tertiary argument in an appeal brief. Given the technical complexity (from the lawyer's viewpoint) of the discussion here and the apparent lack of legal precedent, the lawyer has chosen not to raise it, so at this point the question is mostly theoretical / educational.

To answer one detail: I believe that the number of challenges, 10, was set in advance.

After studying the thoughtful and challenging answers and comments (thanks, all!), it seems that there are 4 separate issues here. For me, at least, it would be most helpful to consider them separately (or to hear arguments why they are not separable.)

1) Is the consideration of the races of both defendant and victims, in the jury pool challenges, of legal concern a priori? The goal of the appeal argument would merely be to raise reasonable concern, which could lead to a judicial order that the prosecution state the reason for each individual challenge. This does not appear to me to be a statistical question, but rather a social / legal one, which is at the lawyer's discretion to raise or not.

2) Assuming (1), is my choice of an alternative hypothesis (qualitatively: bias against jurors who share the defendant's race, in favor of those who share the victims' races) plausible, or is it impermissibly post hoc? From my lay perspective, this is the most perplexing question -- yes, of course one would not raise it if one did not observe it! The problem, as I understand, is selection bias: one's tests should consider not just this jury pool but the universe of all such jury pools, including all the ones where the defense did not observe a discrepancy and therefore were not tempted to raise the issue. How does one address this? (For example, how does Andy's test address this?) It appears, though I may be wrong about this, that most respondents are not troubled by potentially post-hoc 1-tailed tests for bias solely against the defendant's group. How would it be methodologically different to simultaneously test bias for victim groups, assuming (1)?

3) If one stipulates my choice of a qualitative alternative hypothesis as stated in (2), then what is an appropriate statistic for testing it? This is where I am most puzzled by the responses, because the ratio that I propose seems to be a slightly more conservative analog of Andy's test for the simpler "bias against C" alternative hypothesis (more conservative because my test also counts all cases further out in the tail, not just the exact observed count.)

Both tests are simple counting tests, with the same denominator (same universe of samples), and with numerators corresponding precisely to the frequency of those samples which correspond to the respective alternative hypotheses. So @whuber, why is it not identically as true of my counting test as of Andy's that it "can be based on stipulated null [same] and alternative [as described] hypotheses and justified using the Neyman-Pearson lemma"?

4) If one stipulates (2) and (3), are there references in case law which would convince a skeptical appeals court? From the evidence to date, probably not. Also, at this stage of appeal there's no opportunity for any "expert witness", so references are everything.

• Question updated (appended to) after studying answers and comments. Sep 22, 2014 at 0:55
• Thank you for an excellent summary! To respond to point (3), my concern is that your test (if I understand it correctly) adopts an alternative hypothesis that was motivated by the data themselves. It therefore seems to have been constructed a posteriori to make the results appear to be as strong as possible. A test that is based on the broadest possible foreseeable, relevant class of alternatives a priori, and conducted with a Neyman-Pearson rejection region, has a stronger logical foundation and is less subject to criticism that it was nevertheless proposed after seeing the data.
– whuber
Sep 24, 2014 at 16:46
• Thanks, @whuber that's a plausible and helpful criticism -- very much what I was asking about from the start. But wouldn't that cause my (2) to fail, even before (3)? If so, then my (3) would seem to be still unanswered -- i.e. would this a good statistic if one stipulated (2)? Sep 29, 2014 at 1:45

Here's how I might approach answering your question using standard statistical tools.

Below are the results of a probit analysis on the probability of being rejected given the juror's group membership.

First, here's what the data looks like. I have 30 observations of group and a binary rejected indicator:

. tab group rejected

|       rejected
group |         0          1 |     Total
-----------+----------------------+----------
A |         9          1 |        10
B |         6          4 |        10
C |         2          4 |         6
D |         3          1 |         4
-----------+----------------------+----------
Total |        20         10 |        30


Here are the individual marginal effects as well as the joint test:

. qui probit rejected ib2.group

. margins rb2.group

Model VCE    : OIM

Expression   : Pr(rejected), predict()

------------------------------------------------
|         df        chi2     P>chi2
-------------+----------------------------------
group |
(A vs B)  |          1        2.73     0.0986
(C vs B)  |          1        1.17     0.2804
(D vs B)  |          1        0.32     0.5731
Joint  |          3        8.12     0.0436
------------------------------------------------

--------------------------------------------------------------
|            Delta-method
|   Contrast   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
group |
(A vs B)  |        -.3    .181659     -.6560451    .0560451
(C vs B)  |   .2666667   .2470567     -.2175557     .750889
(D vs B)  |       -.15   .2662236     -.6717886    .3717886
--------------------------------------------------------------


Here we are testing the individual hypotheses that the differences in the probability of being rejected for groups A, C, and D compared to group B are zero. If everyone was as likely to be rejected as group B, these would be zero. The last piece of output tells us that group A and D jurors are less likely to be rejected, while group C jurors are more likely to be turned away. These differences are not statistically significant individually, though the signs agree with your bias conjecture.

However, we can reject the joint hypothesis that the three differences are all zero at $p=0.0436$.

If I combine groups A and D into one since they share the victims' races, the probit results get stronger and have a nice symmetry:

Contrasts of adjusted predictions
Model VCE    : OIM

Expression   : Pr(rejected), predict()

------------------------------------------------
|         df        chi2     P>chi2
-------------+----------------------------------
group2 |
(A+D vs B)  |          1        2.02     0.1553
(C vs B)  |          1        1.17     0.2804
Joint  |          2        6.79     0.0336
------------------------------------------------

--------------------------------------------------------------
|            Delta-method
|   Contrast   Std. Err.     [95% Conf. Interval]
-------------+------------------------------------------------
group2 |
(A+D vs B)  |  -.2571429   .1809595      -.611817    .0975313
(C vs B)  |   .2666667   .2470568     -.2175557     .750889
--------------------------------------------------------------


This also allows Fisher's exact to give congruent results (though still not at 5%):

 RECODE of |       rejected
group |         0          1 |     Total
-----------+----------------------+----------
A+D |        12          2 |        14
B |         6          4 |        10
C |         2          4 |         6
-----------+----------------------+----------
Total |        20         10 |        30

Pearson chi2(2) =   5.4857   Pr = 0.064
Fisher's exact =                 0.060

• Thanks, much appreciated! Could you help me understand the methodological issues here? In particular, (1) the undirected comparison tests (IIUC) despite the particularities of the a priori concern, and (2) the reasons to use a test which makes distribution assumptions rather than just combinatorial arguments? Sep 17, 2014 at 11:41
• I am not sure I understand (1). For (2), I get very similar results with a logit model, which makes different distributional assumptions, so there is some robustness. There's not enough data to do something less parametric, though that may be my own ignorance in this area. Sep 17, 2014 at 15:01
• Re (1). What I mean is - it seems that your test is 2-tail, whereas the a priori concern would allow 1-tail? Sep 17, 2014 at 17:04
• An aspect of this analysis that makes me uneasy is that its apparent significance (at the 5% level, anyway) is due not only to the challenges occurring in group C but also to the relative paucity of challenges in group A. The latter would seem to be irrelevant: would it have been suspected a priori? The favored role of group C is evident (in matching the Defendant's group), but a favored role for any other group--or even of (hypothetically) obvious inequities between the other groups--would seem to have no bearing on Defendant's claim of discrimination against them based on their group.
– whuber
Sep 18, 2014 at 22:17
• BTW, it appears you carried out an analysis of group B rather than group C.
– whuber
Sep 18, 2014 at 22:17

I would think that introducing an ad hoc statistical method is going to be a no-go with the court. It is better to use methods that are "standard practice". Otherwise, you probably get to prove your qualifications to develop new methods.

To be more explicit, I do not think that your method would meet the Daubert standard. I also very much doubt that your method has any academic reference in and of itself. You would probably have to go the route of hiring a statistical expert witness to introduce it. It would be easily countered, I would think.

The basic question here is likely: "Was jury challenge independent of racial grouping?"

These are small numbers to which to be applying asymptotically based statistical methods. However, the "standard" for testing association in this setting is the $\chi^2$ test:

> M <- as.table(cbind(c(9, 6, 2, 3), c(1, 4, 4, 1)))
> dimnames(M) <- list(Group=c("A", "B", "C", "D"), Challenged=c("No", "Yes"))
> M
Challenged
Group No Yes
A  9   1
B  6   4
C  2   4
D  3   1

> chisq.test(M)

Pearson's Chi-squared test

data:  M
X-squared = 5.775, df = 3, p-value = 0.1231

Warning message:
In chisq.test(M) : Chi-squared approximation may be incorrect


Using the Fisher exact test gives similar results:

> fisher.test(M)

Fisher's Exact Test for Count Data

data:  M
p-value = 0.1167
alternative hypothesis: two.sided


The note about the hypothesis being two-sided applies to the case of a $2\times2$ table.

My interpretation is that there is not much evidence to argue racial bias.

• Because the $\chi^2$ test is assessing possible differences among all the groups, it is partly measuring differences in challenge rates among groups A, B, and D. The lawyers might argue that groups A and D are distinguished (as being those of victims), but that suggests combining those groups for this test. The resulting p-value increases to 16% (simulated based on a million replications). This bolsters your conclusion (+1).
– whuber
Sep 18, 2014 at 22:23
• Thanks, @jvbraun, your point about ad hoc methods being no-go seems persuasive; though counting and dividing do not seem particularly eccentric to me, clearly others do not find it persuasive! Sep 19, 2014 at 1:12
• This is actually one of the cases in which the marginals are fixed, so Fisher's exact test should be more palatable for many. In your discussion of Daubert you have it a bit backwards, once you call an expert then they are subject to a Daubert motion. (Ironically, some have argued laymen presenting statistics are not subject to such evaluations dictated by Rule 702.) IMO all of the arguments proffered here are well articulated and would be unlikely to be ruled inadmissible. I doubt any of these statistical techniques have jurisprudence in these particular circumstances. Sep 19, 2014 at 14:17
• It wasn't clear to me whether the number of challenges were fixed, but if so then Fisher's exact test is the correct one (given my re-interpretation of the original question). Anything based on large sample sizes or distributional assumptions with n=30 definitely exceeds my comfort level in the legal setting; however, I do believe that the $\chi^2$ test is very commonly applied to this type of question. To me some statistical methods are more applicable to jurisprudence than others, but I have seen some godawful stuff from "the other side". Sep 19, 2014 at 14:56
• I'm sure you could find some examples of $\chi^2$ in some court cases. All the examples I can dig up for Batson challenges though just use simple fractions, e.g. $2/2$, $4/6$. In all states I am familiar with, the total number of peremptory challenges is fixed for each side and is dependent on the severity of the case (i.e. felony cases each side gets more than in misdemeanor cases). Sep 19, 2014 at 15:35

I asked a similar question previously (for reference here is the particular case I discuss). The defense needs to simply show a prima facia case of discrimination in Batson challenges (assuming US criminal law) - so hypothesis tests are probably a larger burden than is needed.

So for:

• $n = 30$ people on the venire panel
• $p = 6$ people of racial class C on the panel
• $k = 4$ jurors of racial group C eliminated on preemptory challenges
• $d = 10$ preemptory challenges

Whuber's previous answer gives the probability of this particular outcome being dictated by the hypergeometric distribution:

$$\frac{{p \choose k} {n-p \choose d-k} }{{n \choose d}}$$

Which Wolfram-Alpha says equals in this case:

$$\frac{{6 \choose 4} {30-6 \choose 10-4} }{{30 \choose 10}} = \frac{76}{1131} \approx 0.07$$

Unfortunately I do not have a reference besides the links I have provided - I imagine you can dig up a suitable reference for the hypergeometric distribution from the Wikipedia page.

This ignores the question about whether racial groups A and D are "under-challenged". I'm skeptical you could make a legal argument for this -- it would be a weird twist on the equal protection clause, This particular group is too protected!, that I do not think would fly. (I'm not a lawyer though - so take with a grain of salt.)

If you really want a hypothesis test I am not sure how to go about it. You can generate the $30 \choose 10$ permutations, give it a probability under the null of racial groups being equally chosen per their proportions in the venire, and then calculate the exact distribution of your test statistic under the null. I'm not quite sure what test statistic is satisfactory though, $\chi^2$ doesn't really answer the question of interest. (Is it alright you make up your own test statistic -- I do not know?)

I've updated some of my thoughts in a blog post. My post is specific to Batson Challenges, so it is unclear if you seek another situation (your updates for 1 and 2 don't make sense in the context of Batson Challenges.)

I was able to find one related article (available in full at the link):

Gastwirth, J. L. (2005). Case comment: statistical tests for the analysis of data on peremptory challenges: clarifying the standard of proof needed to establish a prima facie case of discrimination in Johnson v. California. Law, Probability and Risk, 4(3), 179-185.

That gave the same suggestion for using the hypergeometric distribution. In my blog post I show how if you collapse the categories into two groups it is equivalent to Fisher's Exact test.

Gastwirth suggests as I did in my comment that you could consider $k$ as the test statistic, and so add the probability of $k = 5$ and $k = 6$ to that above if you prefer. Gastwirth also gives an example for calculating a test statistic based on changing numbers of $n$ in the jury pool. In my blog post I just conduct sensitivity analysis for varying levels of $n$ and $d$ (for a different case) to provide ranges of possible percentages.

If someone becomes aware of case law that actually uses this (or anything besides fractions) I would be interested.

• Thanks, Andy. (1) My lawyer friend thinks is perfectly acceptable/useful to assert that C was over-challenged and A under-challenged. (2) You say "what test statistic". I find that confusing -- what test statistic are you using when you compute 0.07 using hypergeometric? What that does is compute the probability as the ratio of suspect cases to total cases. Likewise, that's exactly what my analysis does, except defining suspect cases more narrowly than you do. Sep 18, 2014 at 1:24
• @JonathanMarch - I don't use a test statistic. This is the probability of 4 out of 6 class C being chosen (given the other conditions) randomly according to the hypergeometric distribution. I understand the motivation for directional tests, but this isn't the usual t-test case. In that case you have a continuous null distribution, so to give a p-value you need to define the alternative as an area. There is no implicit need to do that with a PMF distribution as in here. Sep 18, 2014 at 12:09
• If you wanted to, you could add the probability of $k = 5$ and $k = 6$ to the $0.07$ above, but of course that would only increase the probability listed above. Your original calculations is assuming all potential permutations are equally likely. Which I think is defensible, but I believe specifying the data generating process as hypergeometric is more realistic. The partitioning in your question I find intuitive but ad-hoc, I see no reason to interpret it as a probability in any sense. Sep 18, 2014 at 12:22
• (+1)The test statistic is the number of challenges to group C. This is valid and relevant because C could be identified a priori as the race of the defendant. Andy's analysis is perfectly appropriate (and fairly powerful) assuming that 10 peremptory challenges were fixed in advance. I believe (but would need to check) that it's a good approximation assuming the number of peremptory challenges was random. The logic is simple and sweet: if the challenges were assigned randomly to 30 people, what is the chance that 4 or more challenges were made to group C? The answer is $86/1131\approx 7.6\%$.
– whuber
Sep 18, 2014 at 22:11
• Jonathan, for your benefit I'll give you a hard time (just as an opposition expert would). I believe your approach is invalid because you use an ad hoc statistic without theoretical justification; it seems constructed solely to produce a small p-value. Andy's statistic can be based on stipulated null and alternative hypotheses and justified using the Neyman-Pearson lemma. Your statistic seems to be based on a post hoc examination of the results and does not appear to correspond to any alternative hypothesis that would have been asserted prior to (that is, independently) of the voir dire.
– whuber
Sep 19, 2014 at 14:15

Let's not forget the multiple testing issue. Imagine 100 defense lawyers each looking for grounds to appeal. All of the juror rejections had been performed by flipping coins or rolling dice for each prospective juror. Therefore, none of the rejections were racially biased.

Each of the 100 lawyers now does whatever statistical test all of you guys agree on. Roughly five out of that 100 will reject the null hypothesis of "unbiased" and have grounds for appeal.

• IIUC, they would be looking for grounds for the judge to order an examination of the reasons for the each individual rejection. Would it actually be a problem if such an examination occurred in 5 of those 100 cases? Sep 25, 2014 at 14:09