Statistically comparing results from two methods

Question

I am not exactly sure if CrossValidated is the place to ask this question, but decided to give it a shot. This may be thought as an abstract question. I am trying to compare the feasibility of two different methods, let's call them:

Method A
Method B

Both methods take the same parameters and yield a real number greater than 0 as the result:

Method A(parameters) -> a real number greater than 0
Method B(parameters) -> a real number greater than 0

If the result is greater than 1.0, this is a "pass". If the result is less than 1.0, this is a "fail". The closer the number is to 0, the more severe is the situation.

Assuming both methods are reliable, I am trying to figure out which method would give me better results (let's say more "passes").

First, I thought normalizing the results from one method with the other, I could create some kind of index (let's call it i, which would let me achieve the goal mentioned above:

i = Result from Method A / Result from Method B > 1.0 -> Method A gives a better result.

However, using this approach I loose the severity aspect of the problem due to the normalization, meaning:

Example 1:

Result from method A = 5.65
Result from method B = 4.00    
i = Result from method A / Result from Method B = 1.41 (Method A gives a better result)

Example 2:

Result from method A = 1.10
Result from method B = 0.78    
i = Result from method A / Result from Method B = 1.41 (Method A gives a better result)

As seen above both indices yield the same value, 1,41. However, using method A or B does not affect the severity of the result in Example 1 (both > 1.0). Using any method would be fine.

In the meanwhile, using method A in Example 2 is crucial, since it yields a result > 1.0 (pass), whereas the result from method B is < 1.0 (fail).

I am wondering what would be the best way to compare these two methods? A somehow weighted index (this is just intuition, I am not sure how to construct something like this) This question may belong to the field of decision making, however, I am not very knowledgeable about that field (as I am not in statistics, other than basics). Any pointers would be welcome!

Thanks!

PS: I am not sure if I used correct tags for this question. Please let me know if there is a suitable tag, then I can edit and correct them.

Answer 1

The issue here is you're trying to deal with at least two separate questions.

One question is something along the lines of "Is A better than B?" or "Of A and B, which, if any, is better?"

A second question is something like "Does A pass some independent-of-B standard?" or "Of A and B, which of them, if any, pass the independent standard?"

These two questions are not particularly related. A might be markedly better than B but fail the standard badly. A might be markedly worse than B but pass the standard with flying colors.

You can also get situations where you can't distinguish A from B and you can't distinguish B from the pass mark (you can't say it's better or worse on average), but you can distinguish A from the pass mark!

So the basic problem is you can't answer multiple yes-no questions in full generality with a single yes-no analysis.

My advice is to treat the questions separately; "Does A pass", "Does B pass" and "Are A and B distinguishable?" even though the three questions together may not be quite independent; the dependence is low enough that they're all worth answering.

Another approach would be to set up some contrasts (like (A+B)/2 vs pass and A vs B).