I recently got into a debate with a friend of mine because I claimed that the 9th circuit court tends to have its rulings overturned at a greater rate than other courts (this is about statistics trust me!)

Looking at the data, from face value he rejected my claim because "the ninth circuit is only reversed at 10% higher than the rest of the courts".

To keep it short, things got personal and my intelligence was insulted. He asked to see a t-test and ANOVA w/ post hoc tests (even though we both acknowledge its not best suited here) in order for me to make my case.

In my pain I wrote it all up in a document (I wasn't able to attach a file here) but I wanted to share it with all of you for some opinions on the subject.

We also both agreed that "Reversal_Rate" (# of reversals / # of hearings) is the variable of interest here, not the quantity of reversals (since courts with higher number of hearings will have higher reversals by design).

The Problem:

1) So my question is, what are some tests I can use to see if ninth court's Reversal Rate is higher than the rest? In other words, what kind of statistical test is best suited to test proportional data such as Reversal_rt?

2) If someone would be so kind, can you check out my results so far (especially in the second half) and let me know their thoughts?

My results: https://i.sstatic.net/ni4pr.jpg

Data Source: https://ballotpedia.org/SCOTUS_case_reversal_rates_(2007_-_Present)?fbclid=IwAR1rOpTKLg4JRWvZ1Je95Ma0PBURaz5iqJvBu9HpuR3ckjR7WuX2IaB0qu8

  • 2
    $\begingroup$ In order to do a proper test you'd need the numbers of decisions and of reversals for the 9th circuit court and also numbers of decisions and reversals for comparison court(s). Percentages alone are almost useless for testing: 10 reversals for 50 decisions and 100 reversals for 500 decisions carry very different amounts of information. // Answer of @jros is a good place to start. $\endgroup$
    – BruceET
    Commented May 14, 2020 at 7:06
  • $\begingroup$ Is independence of the rulings a reasonable assumption here? $\endgroup$
    – jaradniemi
    Commented May 21, 2020 at 21:19

1 Answer 1


the simplest method for finding evidence of a difference in proportions would be a Chi-Square test of independence. To compare a difference in proportions between two groups, construct a confidence interval in this manner:
$\hat{p_1}$ = proportion for group 1, $n_1$ is sample size of group 1
$\hat{p_2}$ = proportion for group 2, $n_2$ is sample size of group 2
$z^*$ is our critical value, ie for a 95% Confidence Interval, $z^*$ = 1.96

$(\hat{p_1}-\hat{p_2}) \pm z_{1-\frac{\alpha}{2}}^*\sqrt{\frac{\hat{p_1}*(1-\hat{p_1})}{n_1}+\frac{\hat{p_2}*(1-\hat{p_2)}}{n_2}}$

  • $\begingroup$ How would one set that up on a statistical software like Stata? I hesitated trying it because I was worried about the difference in sample size. For instance consider the following: n1: 9th circuit court (20 reversals; 5 affirmations) n2: all other courts (100 reversals 9 affirmations) If I group all other courts into n2, will the difference in sample size matter? Also I would assume that there is no way to do this test using actual proportions (Rate of Reversals). Rather I would have to do it by breaking it down by Reversals vs. Affirmations right? $\endgroup$
    Commented May 14, 2020 at 16:49
  • $\begingroup$ Can you expand on what you mean by actual proportions? For the 9th circuit court, your p-hat would be 20/25 = 0.8; if you have a rate for a really large n, you may be able to treat that as a population standard deviation. $\endgroup$
    – jros
    Commented May 14, 2020 at 21:01
  • 1
    $\begingroup$ Some posts with other/better ways to form CI: stats.stackexchange.com/questions/28934/…, stats.stackexchange.com/questions/400729/…, stats.stackexchange.com/questions/513390/… $\endgroup$ Commented Nov 11, 2021 at 0:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.