A health insurance company here presents rating lists of hospitals based of their SMR (standardized mortality/morbidity ratio) values (observed/expected, i.e. the higher the worse). They use the confidence intervals of SMR as well, which are wide in a small number of cases and narrow in large samples. So far, so good. It follows: they consider the best hospitals as those with SMR below the 20 percentile with lowest values of the upper ends of their confidence intervals and the worst hospitals as those with SMR above the 80 percentile with highest values of the lower ends of their confidence intervals – regardless whether the whole confidence interval of a particular hospital includes 1 or not. As we asked about a rationale for this they argued that the higher value of the lower end of the confidence interval shows the higher probability of the undesirable outcome.
They even give an example. Two hospitals perform the same surgery with 96% success: the hospital A on 500 patients and the hospital B on 50 patients. They mean that the achieving the success value of 96% success is statistically more likely in the hospital A, because their confidence interval is 94 – 98% there. The hospital B would have the confidence interval of 84 to 100% - thus, not that nice.
I always thought that the confidence intervals just show the range where the true value lands with 95% chance (well, for 95% intervals) and that they should be always observed as a whole, not just lower end or upper end. As I read somewhere, confidence interval shows where the bomb lands with the 95% probability, not where the 95% of all the bombs land. For the latter there are tolerance intervals that are calculated differently. In example above there would be no difference between two hospitals on the basis of the mentioned confidence intervals. And it a confidence interval of a SMR includes 1 you cannot claim the result is better or worse as the expected value – regardless how high or low one end of this confidence interval is located.
I’m surprised however since it is a large company with the own statistic department. Please give me an advice whether I interpet the things correctly or miss something.