Joint test of change in variance across matched groups

Question

I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was implemented.

I have about 40 groups of sentences, which are matched on the characteristics of the case. For example, group 1 comprises sentences for people who committed ABH, who were under the influence of drugs, and used a weapon. Group 2 comprises people who committed ABH, who were remorseful, and had no aggravating factors in the case.

For each group, I can further separate sentences outcomes into two groups: those sentenced before a policy was implemented, and those sentenced after the policy was implemented. I can then compare sentenced outcomes in the 'before' and 'after' groups to see whether variability seems to have declined.

I can test each group individually to see whether variability of sentence outcome went up or down, using an F test. However, the results of these tests vary, and I realise that I should really be using a joint test in any case.

I'm also not getting enough power on a group-by-group basis: for most tests, we cannot reject H0, that variability did not change. This is despite the fact that variability has declined for the vast majority of groups, which suggests that if I somehow combine results from all groups, it should be possible to find a statistically significant decline in variance overall.

Could anybody suggest and appropriate test or how I might proceed? I am very grateful for any help.

A few clarifications:

1. I would be happy to assume that amongst a matched group (log) sentence outcomes are drawn from a normal distribution. This assumption will not be perfect, but is probably adequate.
2. The size of matched groups differs. For instance, the group 1 'before' implementation group may have 100 sentences, and the 'group 1 after implementation' group may have 75 sentences. The same figures for group 2 may be 66 and 123.

Answer 1

I would be happy to assume that amongst a matched group (log) sentence outcomes are drawn from a normal distribution. This assumption will not be perfect, but is probably adequate.

The usual F-test for equality of variance is fairly sensitive to non-normality. You may want to consider a more robust measure, or alternatively to use some form of simulation (where you have a more suitable model for the distribution) or resampling (where you don't) of the distribution of the test statistic; resampling could either involve randomization or bootstrapping.

Let me start by discussing, e.g. a Levene test or a Brown-Forsythe test

http://en.wikipedia.org/wiki/Levene%27s_test

http://www.itl.nist.gov/div898/handbook/eda/section3/eda35a.htm

(note the Brown-Forsythe can use trimmed means rather than medians). Levene isn't quite so dependent on the normality assumption as a Bartlett test, but it's still not highly robust.

One standard version of the Levene test is basically just an analysis of variance performed on the absolute deviations of values from the respective group means.

These aren't the only options, however (and sometimes these are criticized for their own assumption of equality of variance of the absolute deviations) but I'm going to stick with them for just a moment while I make a point.

An advantage of this ANOVA on deviations approach is that you can consider more complicated models than a one-way ANOVA, both for the original means and for the deviations themselves.

So imagine you construct first a suitable model for the means (or possibly using trimmed means if there's remaining heavy-tailedness), then conduct some suitable ANOVA for equality of absolute deviations from those means.

There is a version of the Welch procedure that is used for ANOVA that could be applied to this second stage to overcome potential problems with heteroskedasticity.

You don't have to rely on the usual tables however, since you can, as mentioned above, use simulation or resampling for the distribution of the test statistic. For example, the whole two-stage procedure can be performed with a residual bootstrap.

This reduces further both the sensitivity to normality and the issue with the equality of variance assumption at the second stage.