0
$\begingroup$

I have a large database of binary decisions (accept or reject), broken down by state of the applicant, so that for each state I can calculate a proportion of positive decisions. e.g.

New York - 0.21% accepted
Washington - 0.17% accepted
California - 1.25% accepted
Oregon - 2.01% accepted
Alaska - 0.54% accepted
etc...

As you can see, the proportion accepted is sometimes very low, and therefore if I calculate confidence intervals around them (using a binary distribution to bound the intervals between 0 and 1), the interval width can be quite large.

I want a way of determining which states have a particularly high proportion or accepted decisions. As a start, I calculated the overall national proportion, with confidence intervals, and looked whether the confidence intervals of each state overlapped with the overall national confidence intervals. If they didn't I thought this would indicate that state had a proportion significantly higher or lower proportion than average.

However, I'm worried that the overall national proportion might be disproportionately influenced by certain states where a large number of decisions were made.

Also, is my approach a bit simplistic? Should I instead use logistic regression to see whether state is a significant predictor of decision, and if so, do a post-hoc test to see which states differ? If so, what post-hoc test could I do?

Many thanks

$\endgroup$
3
$\begingroup$

Consider two possible approaches:

  1. A random effects model or penalized maximum likelihood estimation. Either of these will shrink effects towards the grand mean proportion in order to increase precision of estimates and help avoid overinterpretation.
  2. Use the bootstrap to get a 0.95 confidence interval for the rank of each proportion across the states. This will provide evidence for separability of the estimates, incorporating the true difficulty of the task. If the state with the highest proportion has a rank of 50 but a confidence interval for the rank of 10-50 you are trying to split hairs.

For individual confidence intervals use the well-performing Wilson interval, which is automatically bounded in $[0,1]$ (see e.g. the R Hmisc binconf function).

$\endgroup$
  • $\begingroup$ Thanks. I've not used a random effects model before, but upon reading it sounds like the right thing to do. Now to figure out how to do it... $\endgroup$ – rw2 May 21 '16 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.