2
$\begingroup$

I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was implemented.

I have about 40 groups of sentences, which are matched on the characteristics of the case. For example, group 1 comprises sentences for people who committed ABH, who were under the influence of drugs, and used a weapon. Group 2 comprises people who committed ABH, who were remorseful, and had no aggravating factors in the case.

For each group, I can further separate sentences outcomes into two groups: those sentenced before a policy was implemented, and those sentenced after the policy was implemented. I can then compare sentenced outcomes in the 'before' and 'after' groups to see whether variability seems to have declined.

I can test each group individually to see whether variability of sentence outcome went up or down, using an F test. However, the results of these tests vary, and I realise that I should really be using a joint test in any case.

I'm also not getting enough power on a group-by-group basis: for most tests, we cannot reject H0, that variability did not change. This is despite the fact that variability has declined for the vast majority of groups, which suggests that if I somehow combine results from all groups, it should be possible to find a statistically significant decline in variance overall.

Could anybody suggest and appropriate test or how I might proceed? I am very grateful for any help.

A few clarifications:

  1. I would be happy to assume that amongst a matched group (log) sentence outcomes are drawn from a normal distribution. This assumption will not be perfect, but is probably adequate.
  2. The size of matched groups differs. For instance, the group 1 'before' implementation group may have 100 sentences, and the 'group 1 after implementation' group may have 75 sentences. The same figures for group 2 may be 66 and 123.
$\endgroup$
1

1 Answer 1

1
$\begingroup$

I would be happy to assume that amongst a matched group (log) sentence outcomes are drawn from a normal distribution. This assumption will not be perfect, but is probably adequate.

The usual F-test for equality of variance is fairly sensitive to non-normality. You may want to consider a more robust measure, or alternatively to use some form of simulation (where you have a more suitable model for the distribution) or resampling (where you don't) of the distribution of the test statistic; resampling could either involve randomization or bootstrapping.

Let me start by discussing, e.g. a Levene test or a Brown-Forsythe test

http://en.wikipedia.org/wiki/Levene%27s_test

http://www.itl.nist.gov/div898/handbook/eda/section3/eda35a.htm

(note the Brown-Forsythe can use trimmed means rather than medians). Levene isn't quite so dependent on the normality assumption as a Bartlett test, but it's still not highly robust.

One standard version of the Levene test is basically just an analysis of variance performed on the absolute deviations of values from the respective group means.

These aren't the only options, however (and sometimes these are criticized for their own assumption of equality of variance of the absolute deviations) but I'm going to stick with them for just a moment while I make a point.

An advantage of this ANOVA on deviations approach is that you can consider more complicated models than a one-way ANOVA, both for the original means and for the deviations themselves.

So imagine you construct first a suitable model for the means (or possibly using trimmed means if there's remaining heavy-tailedness), then conduct some suitable ANOVA for equality of absolute deviations from those means.

There is a version of the Welch procedure that is used for ANOVA that could be applied to this second stage to overcome potential problems with heteroskedasticity.

You don't have to rely on the usual tables however, since you can, as mentioned above, use simulation or resampling for the distribution of the test statistic. For example, the whole two-stage procedure can be performed with a residual bootstrap.

This reduces further both the sensitivity to normality and the issue with the equality of variance assumption at the second stage.

$\endgroup$
4
  • $\begingroup$ Having tried to implement a Levene test today, I've realized that I don't think it's what I'm looking for. It extends the F test by allowing a comparison of equality among several samples. If the F test allows a comparison of the variance of A vs B, the Levene test allows a comparison of A vs B vs C vs D etc. However, what I'm after is a test of ((A1 vs B1) and (A2 vs B2) and (A3 vs B3)) etc. to see whether the As have difference variance to the Bs. I've been googling for this and haven't been able to turn anything up. Does anyone have any ideas of how I might achieve this joint test? $\endgroup$
    – RobinL
    Apr 2, 2013 at 13:35
  • $\begingroup$ The plain Levene won't - but the approach it uses can be adapted to a more complex model ... as I explained in my answer. Perhaps you should re-read it more carefully. $\endgroup$
    – Glen_b
    Apr 2, 2013 at 13:39
  • $\begingroup$ Thanks for the quick response. So you think with further exploration it would be possible to derive a test statistic even though I don't have any information about the relationship among the As or Bs (e.g. A1 and A2 may have difference variance and mean). $\endgroup$
    – RobinL
    Apr 2, 2013 at 13:43
  • $\begingroup$ The original suggestion was that you would use the particular ANOVA-like model on the absolute deviations which would make the comparison of interest a 'standard' test you'd do with ANOVA (whether or not you actually use the usual F tables being then a second issue). It may help to think in terms of comparing means and what test in ANOVA you'd use for that; the variance comparison you want is then the same test based off the absolute deviations from an appropriate model for the mean. $\endgroup$
    – Glen_b
    Apr 2, 2013 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.