8
$\begingroup$

First off let me say that I had one stats course in engineering school 38 years ago. So I'm flying blind here.

I've got the results of what are essentially 18 separate diagnostic tests for a disease. Each test is binary -- yes/no, with no threshold that can be adjusted to "tune" the test. For each test I have what is ostensibly valid data on true/false positives/negatives when compared to the "gold standard", yielding specificity and sensitivity numbers (and anything else you can derive from that data).

Of course, no single test has sufficient specificity/sensitivity to be used alone, and when you "eyeball" the results of all tests there's frequently no obvious trend.

I'm wondering what is the best way to combine these numbers in a way that will yield a final score that is (hopefully) more reliable than any single test. I've so far come up with the technique of combining of the specificities of TRUE tests using

spec_combined = 1 - (1 - spec_1) * (1 - spec_2) * ... (1 - spec_N)

and combining sensitivities of the FALSE tests the same way. The ratio

(1 - sens_combined) / (1 - spec_combined)

then seems to yield a reasonably good "final score", with a value over 10 or so being a reliable TRUE and a value under 0.1 or so being a reliable FALSE.

But this scheme lacks any true rigor, and for some combinations of test results it seems to produce an answer that is counter-intuitive.

Is there a better way to combine the test results of multiple tests, given their specificities and sensitivities? (Some tests have a specificity of 85 and sensitivity of 15, other tests are just the opposite.)

OK, my head hurts!

Let's say I've got tests 1-4 with sensitivities/specificities (in %):

  1. 65/50
  2. 25/70
  3. 30/60
  4. 85/35

Tests 1 and 2 are positive, 3 and 4 negative.

The putative probability that 1 is a false positive would be (1 - 0.5), and for 2 (1 - 0.7), so the probability that both are false positives would be 0.5 x 0.3 = 0.15.

The putative probability that 3 and 4 are false negatives would be (1 - 0.3) and (1 - 0.85) or 0.7 x 0.15 = 0.105.

(We'll ignore for the moment the fact that the numbers don't add up.)

But the presumed probabilities that 1 and 2 are true positives are 0.65 and 0.25 = 0.1625, while the presumed probabilities that 3 and 4 are true negatives are 0.6 and 0.35 = 0.21.

Now we can ask two questions:

  1. Why don't the numbers add up (or even come close). (The sens/spec numbers I used are from "real life".)
  2. How should I decide which hypothesis is (most likely) true (in this example it seems to be "negative" for both calcs, but I'm not sure that's always the case), and what can I use for a "figure of merit" to decide if the result is "significant"?

More info

This is an attempt to refine and extend an existing "weighting" scheme that is entirely "artistic" in nature (ie, just pulled out of someone's a**). The current scheme is basically on the lines of "If any two of the first three are positive, and if two of the next four, and either of the next two, then assume positive." (That's a somewhat simplified example, of course.) The available statistics don't support that weighting scheme -- even with a crude weighting algorithm based on the measured stats I come up with significantly different answers. But, absent a rigorous way of evaluating the stats I have no credibility.

Also, the current scheme only decides positive/negative, and I need to create a (statistically valid) "ambiguous" case in the middle, so some figure of merit is needed.

Latest

I've implemented a more-or-less "pure" Bayesian inference algorithm, and, after going round and round on several side issues, it seems to be working pretty well. Rather than working from specificities and sensitivities I derive the formula inputs directly from the true positive/false positive numbers. Unfortunately, this means that I can't use some of the better quality data that isn't presented in a way that allows these numbers to be extracted, but the algorithm is much cleaner, allows modification of the inputs with much less hand calculation, and it seems pretty stable and the results match "intuition" fairly well.

I've also come up with an "algorithm" (in the purely programming sense) to handle the interactions between interdependent observations. Basically, rather that looking for a sweeping formula, instead I keep for each observation a marginal probability multiplier that is modified as earlier observations are processed, based on a simple table -- "If observation A is true then modify observation B's marginal probability by a factor of 1.2", eg. Not elegant, by any means, but serviceable, and it seems to be reasonably stable across a range of inputs.

(I'll award the bounty to what I deem to have been the most helpful post in a few hours, so if anyone wants to get a few licks in, have at it.)

Improve this question
$\endgroup$
9
  • $\begingroup$ The probability that test 1 is a false positive is not 1 - .5, it's 1 - (.5 * Probability of Not having the disease) $\endgroup$
    – fgregg
    Commented Jul 17, 2011 at 22:04
  • $\begingroup$ Good point. That may help me make a bit more sense of things. $\endgroup$
    – Daniel R Hicks
    Commented Jul 18, 2011 at 1:54
  • $\begingroup$ Sorry, actually, I was wrong. Specificity = Pr(True Negative)/[Pr(True Negative) + Pr(False Positive)] so Pr(False Positive) = Pr(True Negative)/specificity - Pr(True Negative) which equals Pr(False Positive) = Pr(No Disease)(1 - specificity) $\endgroup$
    – fgregg
    Commented Jul 18, 2011 at 3:41
  • 1
    $\begingroup$ Just to clarify: when you say you are looking for rigor, you don't mean "statistical rigor", i.e. you don't necessarily need the combined test to give you accurate probabilities of type 1 and 2 errors, right? You are just looking for something that's not pulled out of thin air? $\endgroup$
    – SheldonCooper
    Commented Jul 22, 2011 at 3:01
  • 1
    $\begingroup$ How do you know that the tests are strongly interdependent? Is it your a priori high level knowledge (e.g. both tests use blood pressure, so probably are correlated), or do you have statistics that show that they are correlated? If the latter, you can use a slight modification of fgregg's proposal: model all tests as independent, except for those interdependent pairs, which you should model as pairs. This will require some extra statistics (of the form $p(T_1, T_2 | Disease)$), which you presumably have since you know they are correlated. $\endgroup$
    – SheldonCooper
    Commented Jul 22, 2011 at 4:15

2 Answers 2

Reset to default
1
$\begingroup$

"I'm wondering what is the best way to combine these numbers in a way that will yield a final score that is (hopefully) more reliable than any single test." A very common way is to compute Cronbach's alpha and, more generally, to perform what some would call a "standard" reliability analysis. This would show to what degree a given score correlates with the mean of the 17 other scores; which tests' scores might be best dropped from the scale; and what the internal consistency reliability is both with all 18 and with a given subset. Now, some of your comments seem to indicate that many of these 18 are uncorrelated; if that is true, you may end up with a scale that consists of just a few tests.

EDIT AFTER COMMENT: Another approach draws on the idea that there is a tradeoff between internal consistency and validity. The less correlated your tests are, the better their content coverage, which enhances content validity (if not reliability). So thinking along these lines you would ignore Cronbach's alpha and the related indicators of item-total correlation and instead use a priori reasoning to combine the 18 tests into a scale. Hopefully such a scale would correlate highly with your gold standard.

Improve this answer
$\endgroup$
1
  • $\begingroup$ For various reasons (basically conservative medical bias) I don't have the option of eliminating any tests, nor do I especially want to. Think of it as maybe analogous to a credit score, where having a big credit card debt is "uncorrelated" with having a poorly-paying, apt-to-be-laid-off job, but the two together create a much more serious situation than either individually. $\endgroup$
    – Daniel R Hicks
    Commented Jul 16, 2011 at 16:16
1
+100
$\begingroup$

To simplify a bit, let's assume that you only have two diagnostic tests. You want to calculate

$$ \Pr(\text{Disease} \mid T_1,T_2) = \frac{\Pr(T_1,T_2 \mid \text{Disease})\Pr(\text{Disease})}{\Pr(T_1,T_2)} $$

You suggested that the results of these tests are independent, conditional on person having a disease. If so, then

$$ \Pr(T_1,T_2 \mid \text{Disease}) = \Pr(T_1 \mid \text{Disease})\Pr(T_2 \mid \text{Disease}) $$

Where $\Pr(T_i \mid \text{Disease})$ is the sensitivity of Test $i$.

$\Pr(T_1,T_2)$ is the unconditional probability of a random person testing positive on both tests:

$$ \Pr(T_1,T_2) = \Pr(T_1,T_2 \mid \text{Disease})\Pr(\text{Disease}) + \Pr(T_1,T_2 \mid \text{No Disease})\Pr(\text{No Disease}) $$

Where

$$ \Pr(T_1,T_2 \mid \text{No Disease}) = \Pr(T_1 \mid \text{No Disease})\Pr(T_2 \mid \text{No Disease}) $$

and $\Pr(T_i \mid \text{No Disease})$ is $1 - \text{specificity}$ for Test $i$.

Improve this answer
$\endgroup$
12
  • $\begingroup$ I'm not sure this works in my case (if I'm understanding "logistic regression" halfway correctly). First off, as described, there are no (or at least relatively few) thresholds or tuning factors I can adjust for each individual test -- just positive/negative results. Secondly, I don't have the luxury of being able to obtain new data to "train" the model -- just coming up with the data I do have was like pulling teeth. $\endgroup$
    – Daniel R Hicks
    Commented Jul 14, 2011 at 14:34
  • $\begingroup$ Could you describe your data a bit more. I thought that you knew the ground truth of whether the cases had the disease or not? $\endgroup$
    – fgregg
    Commented Jul 14, 2011 at 14:55
  • 1
    $\begingroup$ The difficulty you have with the numbers not matching up is with redundant information. For instance suppose one of the test is "systolic blood pressure (SBP) > 140", and the other is "diastolic blood pressure (DBP) > 90". Well these 2 are correlated and the information inherent in each is not unique. Combining them logically, say "SBP>140 or DBP>90" will offer incremental improvement in sensitivity. But without a dataset that simultaneously measured the gold standard, SBP, and DBP, there is no accurate way to quantify sensitivity and specificity of the combined test. $\endgroup$
    – Ming K
    Commented Jul 16, 2011 at 21:31
  • 1
    $\begingroup$ @Daniel: It seems like you won't need any new data (beyond what you already have) for this approach. It seems you will need the true/false positive/negative rates, and you don't need any thresholds. $\endgroup$
    – SheldonCooper
    Commented Jul 22, 2011 at 3:10
  • 1
    $\begingroup$ @Daniel: this was in response to your comment from Jul 14. What fgregg has described is basically a Naive Bayes approach. It seems that you have enough information to use this approach. All you need is the rates, which you have. You don't need any new information, and you don't need any thresholds on the tests. It seems you figured this out already, since you say you tried it. You are right that any dependencies will skew the results. $\endgroup$
    – SheldonCooper
    Commented Jul 22, 2011 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.