How can I compare two classifications?

Question

My job is to develop a classification that groups together diagnoses of similar cost and procedures of similar cost, by similar anatomy. For instance a hip replacement would not be in the same group as a heart transplant as they cost different amounts and are not clinically similar. We revise this huge classification yearly and I've been tasked to prove that changes we make, make it 'better'. I know :-( Numerically we used to look at reduction in variance, does this way of grouping explain more variance than this way of grouping? However, a lot of the clustering decisions are based on clinical reasoning, not numerical. Two procedures may cost the same but you may not put them in the same group as one requires anaesthetic and one doesn't! This obviously cannot be measured in the same way. I'm not wanting to assess the accuracy of the algorithm (ie has the patient dropped into the right group for what is wrong with them) but a method of comparing the way groups are defined. Thanks!!

I couldn't add a comment to your question so I've added here:

The classification is used to collect costs from hospitals. Each hospital calculates the average cost to them to deliver the service described by each group within the classification (there are around 1500 groups within the classification such as Heart transplants, disorders of spleen, procedures on lymphatic system etc..). These hospital level costs are collated to provide a national average for each group which in turn is used to calculate a tariff for the government to pay those hospitals for that service. The variance I described is the variance in costs reported - one hospital may say it costs them £2000 to do a hip replacment, another may say £8000. If we say there are 20 different ways to define a hip replacement (different kinds of surgery), 19 of the 20 may cost around 2K while one may cost £20k. It would be appropriate to move that 20K hip replacement into a different group with other 20K procedures. How could we show statistically that making this change has improved the quality of the costs within each group by reducing the cost variance in each group? Hope that makes sense!

Answer 1

In your Nov. 16 comment you say "How would I know which classification is 'best' using cost of each group as the measure." This seems to simplify matters dramatically and perhaps to make unnecessary some of the considerations described in your original question. If the main thing you want to do is to compare the success of each clustering arrangement in explaining the variance in cost, that would in most cases be pretty straightforward. Since cost is a continuous variable and group is a nominal/categorical one, you could ordinarily run an ANOVA to see to what extent the clustering accounts for cost.

The problem is that you refer to 1500 groups. If this is the end-result of clustering, then you've got your cases much too splintered to deal with in a statistical test (unless you are dealing with vast amounts of cases--say, from the entire U.S.). You've got a statistical power problem. So if I have understood you correctly the 1500 groups may themselves need to be reclustered into just a few before you can formally test their performance in accounting for the variance in cost. Just how few would be ascertainable through a power analysis that drew on the typical mean differences among the final groups; the typical amount of variance within each group; the N-size within each group; and your threshold for statistical significance. Assuming you got the reclustering done (no small feat), such a power analyis could be facilitated using the free program GPower.