I am not exactly sure if CrossValidated is the place to ask this question, but decided to give it a shot. This may be thought as an abstract question. I am trying to compare the feasibility of two different methods, let's call them:
Method A Method B
Both methods take the same parameters and yield a real number greater than 0 as the result:
Method A(parameters) -> a real number greater than 0 Method B(parameters) -> a real number greater than 0
If the result is greater than 1.0, this is a "pass". If the result is less than 1.0, this is a "fail". The closer the number is to 0, the more severe is the situation.
Assuming both methods are reliable, I am trying to figure out which method would give me better results (let's say more "passes").
First, I thought normalizing the results from one method with the other, I could create some kind of index (let's call it
i, which would let me achieve the goal mentioned above:
i = Result from Method A / Result from Method B > 1.0 -> Method A gives a better result.
However, using this approach I loose the severity aspect of the problem due to the normalization, meaning:
Result from method A = 5.65 Result from method B = 4.00 i = Result from method A / Result from Method B = 1.41 (Method A gives a better result)
Result from method A = 1.10 Result from method B = 0.78 i = Result from method A / Result from Method B = 1.41 (Method A gives a better result)
As seen above both indices yield the same value, 1,41. However, using method A or B does not affect the severity of the result in Example 1 (both > 1.0). Using any method would be fine.
In the meanwhile, using method A in Example 2 is crucial, since it yields a result > 1.0 (pass), whereas the result from method B is < 1.0 (fail).
I am wondering what would be the best way to compare these two methods? A somehow weighted index (this is just intuition, I am not sure how to construct something like this) This question may belong to the field of decision making, however, I am not very knowledgeable about that field (as I am not in statistics, other than basics). Any pointers would be welcome!
PS: I am not sure if I used correct tags for this question. Please let me know if there is a suitable tag, then I can edit and correct them.