1
$\begingroup$

A health insurance company here presents rating lists of hospitals based of their SMR (standardized mortality/morbidity ratio) values (observed/expected, i.e. the higher the worse). They use the confidence intervals of SMR as well, which are wide in a small number of cases and narrow in large samples. So far, so good. It follows: they consider the best hospitals as those with SMR below the 20 percentile with lowest values of the upper ends of their confidence intervals and the worst hospitals as those with SMR above the 80 percentile with highest values of the lower ends of their confidence intervals – regardless whether the whole confidence interval of a particular hospital includes 1 or not. As we asked about a rationale for this they argued that the higher value of the lower end of the confidence interval shows the higher probability of the undesirable outcome.

They even give an example. Two hospitals perform the same surgery with 96% success: the hospital A on 500 patients and the hospital B on 50 patients. They mean that the achieving the success value of 96% success is statistically more likely in the hospital A, because their confidence interval is 94 – 98% there. The hospital B would have the confidence interval of 84 to 100% - thus, not that nice.

I always thought that the confidence intervals just show the range where the true value lands with 95% chance (well, for 95% intervals) and that they should be always observed as a whole, not just lower end or upper end. As I read somewhere, confidence interval shows where the bomb lands with the 95% probability, not where the 95% of all the bombs land. For the latter there are tolerance intervals that are calculated differently. In example above there would be no difference between two hospitals on the basis of the mentioned confidence intervals. And it a confidence interval of a SMR includes 1 you cannot claim the result is better or worse as the expected value – regardless how high or low one end of this confidence interval is located.

I’m surprised however since it is a large company with the own statistic department. Please give me an advice whether I interpet the things correctly or miss something.

Thank you!

$\endgroup$
1
$\begingroup$

Your understanding of confidence intervals is a little flawed - they are not where the true value lands 95% of the time - its where, if we repeatedly resampled the population and conducted the same study, 95% of the estimates would fall.

There's no reason the lower or upper bound of a confidence interval couldn't be considered separately for some questions - you implicitly do so in this part of your question: "regardless whether the whole confidence interval of a particular hospital includes 1 or not", which in essence is the same as "Is the lower bound below 1 for estimates above one?" and "Is the upper bound above one for estimates below 1". That kind of ad hoc significance test is invariably only considering one of the two bounds.

As for the meat of your question, there are definitely some issues with using the confidence intervals the way they are, and I'm a little skeptical of their approach. It inherently penalizes small hospitals, as they have wider confidence intervals by virtue of their size and the number of events that occur. This may be an overcorrection for a number of rating schemes that penalize the opposite direction, favoring small hospitals (Consumer Reports, I'm looking at you). More generally, I'm skeptical about rating hospitals against each other generally.

$\endgroup$
  • $\begingroup$ Thank you very much! I thought that large hospitals could be penalized as well: consider SMR 0.95 (0,65;1,25) and 1,05 (0,55;1,55) - with the above approach the first one would be (and is indeed) declared worse since the lower bound is higher. I would say that in both the results are not significantly different from expected value and are not different among themselves. I wanted to object their reading of higher lower bound as a proof of higher probability of undesirable outcome. Your remark in the first section of your answer resp. understanding of conf. intervals is of course correct. $\endgroup$ – exp123 Jan 29 '16 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.