First off let me say that I had one stats course in engineering school 38 years ago. So I'm flying blind here.
I've got the results of what are essentially 18 separate diagnostic tests for a disease. Each test is binary -- yes/no, with no threshold that can be adjusted to "tune" the test. For each test I have what is ostensibly valid data on true/false positives/negatives when compared to the "gold standard", yielding specificity and sensitivity numbers (and anything else you can derive from that data).
Of course, no single test has sufficient specificity/sensitivity to be used alone, and when you "eyeball" the results of all tests there's frequently no obvious trend.
I'm wondering what is the best way to combine these numbers in a way that will yield a final score that is (hopefully) more reliable than any single test. I've so far come up with the technique of combining of the specificities of TRUE tests using
spec_combined = 1 - (1 - spec_1) * (1 - spec_2) * ... (1 - spec_N)
and combining sensitivities of the FALSE tests the same way. The ratio
(1 - sens_combined) / (1 - spec_combined)
then seems to yield a reasonably good "final score", with a value over 10 or so being a reliable TRUE and a value under 0.1 or so being a reliable FALSE.
But this scheme lacks any true rigor, and for some combinations of test results it seems to produce an answer that is counter-intuitive.
Is there a better way to combine the test results of multiple tests, given their specificities and sensitivities? (Some tests have a specificity of 85 and sensitivity of 15, other tests are just the opposite.)
OK, my head hurts!
Let's say I've got tests 1-4 with sensitivities/specificities (in %):
Tests 1 and 2 are positive, 3 and 4 negative.
The putative probability that 1 is a false positive would be (1 - 0.5), and for 2 (1 - 0.7), so the probability that both are false positives would be 0.5 x 0.3 = 0.15.
The putative probability that 3 and 4 are false negatives would be (1 - 0.3) and (1 - 0.85) or 0.7 x 0.15 = 0.105.
(We'll ignore for the moment the fact that the numbers don't add up.)
But the presumed probabilities that 1 and 2 are true positives are 0.65 and 0.25 = 0.1625, while the presumed probabilities that 3 and 4 are true negatives are 0.6 and 0.35 = 0.21.
Now we can ask two questions:
- Why don't the numbers add up (or even come close). (The sens/spec numbers I used are from "real life".)
- How should I decide which hypothesis is (most likely) true (in this example it seems to be "negative" for both calcs, but I'm not sure that's always the case), and what can I use for a "figure of merit" to decide if the result is "significant"?
This is an attempt to refine and extend an existing "weighting" scheme that is entirely "artistic" in nature (ie, just pulled out of someone's a**). The current scheme is basically on the lines of "If any two of the first three are positive, and if two of the next four, and either of the next two, then assume positive." (That's a somewhat simplified example, of course.) The available statistics don't support that weighting scheme -- even with a crude weighting algorithm based on the measured stats I come up with significantly different answers. But, absent a rigorous way of evaluating the stats I have no credibility.
Also, the current scheme only decides positive/negative, and I need to create a (statistically valid) "ambiguous" case in the middle, so some figure of merit is needed.
I've implemented a more-or-less "pure" Bayesian inference algorithm, and, after going round and round on several side issues, it seems to be working pretty well. Rather than working from specificities and sensitivities I derive the formula inputs directly from the true positive/false positive numbers. Unfortunately, this means that I can't use some of the better quality data that isn't presented in a way that allows these numbers to be extracted, but the algorithm is much cleaner, allows modification of the inputs with much less hand calculation, and it seems pretty stable and the results match "intuition" fairly well.
I've also come up with an "algorithm" (in the purely programming sense) to handle the interactions between interdependent observations. Basically, rather that looking for a sweeping formula, instead I keep for each observation a marginal probability multiplier that is modified as earlier observations are processed, based on a simple table -- "If observation A is true then modify observation B's marginal probability by a factor of 1.2", eg. Not elegant, by any means, but serviceable, and it seems to be reasonably stable across a range of inputs.
(I'll award the bounty to what I deem to have been the most helpful post in a few hours, so if anyone wants to get a few licks in, have at it.)