Analysing partially ordered responses

Question

I'd like to analyse judges' decisions on bail. Ie some decisions (remand in prison) and clearly worse than others (unconditional bail). But in the middle there are a range of possible responses that are not necessarily better or worse (eg, pay deposit v report to the police on a regular basis v a curfew).

The decisions will include a collection of these options (eg an accused must pay deposit AND report to the police).

I'd like to look for patterns, ie if one judge consistently makes one collection of orders rather than another.

EDIT: I'm thinking of treating them as ordinal as follows. Eg, say the options are:

• Remand in prison
• Release on conditional bail, conditions to include a. deposit, b. curfew, c. reporting.
• Release on unconditional bail

Then I could order them as follows:

1. Remand in prison
2. Release with 3 conditions
3. Release with 2 conditions
4. Release with 1 condition
5. Unconditional release

EDIT2: Another option might be to adapt Will C's approach below and treat them as nominal. Prison and unconditional bail could have their own code. Then the combinations of the conditional bail orders could be coded. Ie:

1. Remand in prison
2. Deposit
3. Reporting
4. Curfew
5. Deposit + Reporting
6. Deposit + Curfew
7. etc
8. Deposit + Reporting + Curfew
9. Unconditional release

The downside of this is that you would lose the ordering of some decisions being 'better' than others.

Answer 1

If your concern is testing whether or not a judge makes a certain type of decision more than the other judge makes that type of decision, a series of chi-square tests or Fisher exact tests may be your best bet. However you might have to organize your data a bit before you can immediately apply this.

You could start by creating a contingency table for each judge and test, following the advice here, with rows describing each decision and columns for both the observed number of times that decision was made for the judge, and the expected number of times for that decision (i.e. probability of making that decision at random * the number of decisions made total), as well as the same columns for times that decision was not made.

We then could compute a chi-square test statistic by summing up (observed - expected)^2 / total for each decision. The resulting test would tell us whether or not for that group of tests the judge was more likely to make that decision than random. if the number of decisions is small, you could also use Fisher's exact test here.

You could also extend this to groups of decisions for each judge and look into whether certain judges are more likely than not to make certain categories of decisions.