4
$\begingroup$

I need to compare the ability of two methods to predict an event with a binary response. Each method produces a score, where the higher score indicates 1 and a lower score indicates 0. I am looking to calculate the odds ratio (in univariable and multivariable analysis) to compare these methods. Here is the break down of method data:

Method 1: Probability between 0 and 1.

Method 2: Score between 0 and 12

In univariable analysis, the OR for method 1 will be VERY large, but for method 2 would be quite small. I think that because the two methods have different scales the odds ratios become so vastly different. The unit increase for method one should be 0.1, whereas for method two is 1.

Possible solutions are to binarize both methods, by picking a cutoff, then everything is on the 0 and 1 scale and the OR are comparable, however you loose information when you do this. Also, it can be trickey to pick an objective cutoff.

Are there any suggestions for comparing the odds ratios of continous and discrete variables?

Improve this question
$\endgroup$
1
  • $\begingroup$ If I multiplied the probabilities in method 1 by 100, would the "unit increase" result in the same interpretation when comparing the OR of the two methods? There is still the issue that one is disrecete and one is continuous, but at least the unit increase is 1 in both cases, not 0.1. The range would still be very different, and a 5 from method 1 means something very different in Method 2 (in terms of severity), how much does that matter? $\endgroup$
    – user4673
    Commented Mar 2, 2012 at 21:09

2 Answers 2

Reset to default
1
$\begingroup$

Are the measures actually measuring the same thing?

For example, does a score of 0 in Method 1 match a score of 1 in Method 2, and a score of 1 match a score of 12?

If so, you can take exp(beta*12), where beta is the regression coefficient from your logistic regression for Method 2. That would essentially give you the odds ratio not for a single step increase (1 to 2, 2 to 3, etc.) but the odds ratio for an increase from one end of the scale to the other.

If instead say a 0 in Method 1 is probably closer to a 2 in Method 2, and a score of 1 is more like a 10, then you'd multiply by 10-2 = 8 instead of 12.

That should get you a more comparable interval. There are other methods you can try as well, depending on the distribution of Method 2's scores. This retains much of the information from Method 2 while still providing something approaching a like-with-like comparison.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Can you please advise how I would determine: "does a score of 0 in Method 1 match a score of 1 in Method 2, and a score of 1 match a score of 12?". My immediate answer is : "Yes". Low scores in one method mean the same thing as low scores in the other method and vice versa. But, how do I determine what range of proabilities equals a score? For example, does 0.41 to 0.5 translate to a score of 6? $\endgroup$
    – user4673
    Commented Feb 22, 2012 at 17:32
  • $\begingroup$ I would like to add one more point: Method 2 is overly pessimistic. So, the same patient would receive a low score via method1 and a high score via method 2. Based on a ROC, method1 does better overall than method2 (but I still want to use OR as another point of comparison). Based on the difference in the distribution between the two methods, how might this impact the comparison you've proposed. $\endgroup$
    – user4673
    Commented Feb 22, 2012 at 18:08
  • 1
    $\begingroup$ @user4673 There are many ways to determine "match", based on ytour scales. Some scales are matched - validation studies or the original design made them so. You could also look at the distirubtion of your continuous score to see if there are two clear "humps". With that, you're essentially calculating the odds ratio of heading from the bottom of the distribution to the top (which is similar, but not identical, to going from 0 to 1 in the binary score). Honestly, more specific advice beyond that is probably best left to one who knows your data - any chance you have access to a stats type? $\endgroup$
    – Fomite
    Commented Feb 23, 2012 at 0:24
0
$\begingroup$

What I ended up doing that seemed simplest was I categorized the data into quartiles. I still loose information, but hopefully a little less than binarizing the data. Now the two methods are on the same scale and each increment in the scale has (I think) the same meaning.

I was also advised that I may want to look at the OR of going from one quartile to the next and doing the comparison that way. So compared the OR for going from Q1 to Q2 in method A against Q1 to Q2 in method B. So this way I could really compared how each method was performing.

I think I could also use something more granular than quartiles, and just replace the scores in methods A and B with the decile results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.