I am not exactly sure if CrossValidated is the place to ask this question, but decided to give it a shot. This may be thought as an abstract question. I am trying to compare the feasibility of two different methods, let's call them:

Method A
Method B

Both methods take the same parameters and yield a real number greater than 0 as the result:

Method A(parameters) -> a real number greater than 0
Method B(parameters) -> a real number greater than 0

If the result is greater than 1.0, this is a "pass". If the result is less than 1.0, this is a "fail". The closer the number is to 0, the more severe is the situation.

Assuming both methods are reliable, I am trying to figure out which method would give me better results (let's say more "passes").

First, I thought normalizing the results from one method with the other, I could create some kind of index (let's call it i, which would let me achieve the goal mentioned above:

i = Result from Method A / Result from Method B > 1.0 -> Method A gives a better result.

However, using this approach I loose the severity aspect of the problem due to the normalization, meaning:

Example 1:

Result from method A = 5.65
Result from method B = 4.00    
i = Result from method A / Result from Method B = 1.41 (Method A gives a better result)

Example 2:

Result from method A = 1.10
Result from method B = 0.78    
i = Result from method A / Result from Method B = 1.41 (Method A gives a better result)

As seen above both indices yield the same value, 1,41. However, using method A or B does not affect the severity of the result in Example 1 (both > 1.0). Using any method would be fine.

In the meanwhile, using method A in Example 2 is crucial, since it yields a result > 1.0 (pass), whereas the result from method B is < 1.0 (fail).

I am wondering what would be the best way to compare these two methods? A somehow weighted index (this is just intuition, I am not sure how to construct something like this) This question may belong to the field of decision making, however, I am not very knowledgeable about that field (as I am not in statistics, other than basics). Any pointers would be welcome!


PS: I am not sure if I used correct tags for this question. Please let me know if there is a suitable tag, then I can edit and correct them.


1 Answer 1


The issue here is you're trying to deal with at least two separate questions.

One question is something along the lines of "Is A better than B?" or "Of A and B, which, if any, is better?"

A second question is something like "Does A pass some independent-of-B standard?" or "Of A and B, which of them, if any, pass the independent standard?"

These two questions are not particularly related. A might be markedly better than B but fail the standard badly. A might be markedly worse than B but pass the standard with flying colors.

You can also get situations where you can't distinguish A from B and you can't distinguish B from the pass mark (you can't say it's better or worse on average), but you can distinguish A from the pass mark!

So the basic problem is you can't answer multiple yes-no questions in full generality with a single yes-no analysis.

My advice is to treat the questions separately; "Does A pass", "Does B pass" and "Are A and B distinguishable?" even though the three questions together may not be quite independent; the dependence is low enough that they're all worth answering.

Another approach would be to set up some contrasts (like (A+B)/2 vs pass and A vs B).

  • $\begingroup$ thanks for the answer! It looks like the best course would be treating the questions separately, as you said. Maybe I can split the results from Method A into two (>1.0 and <1.0) then calculate indices and make comparisons independently for each group. I had just wondered if there is a more elegant method to accomplish this. $\endgroup$
    – marillion
    May 20, 2013 at 15:03
  • $\begingroup$ What question would such a split answer? $\endgroup$
    – Glen_b
    May 20, 2013 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.