I'm dealing with a problem I've never tackled before. I have to compare two diagnostic tests, which are used to predict the presence of a certain number of conditions, all belonging to the same family. I'm interested in evaluating the test independently of the specific condition, i.e. I generally want to know how good it is in predicting conditions belonging to this family. I have cases where one of the conditions is predicted and actually observed, cases where the predicted conditions is then observed together with other non predicted ones, cases in which no conditions are predicted and one or more are observed...the situation goes something like this:
instance | predicted | observed ----------------------------------------------- 1 | condA, condB | condB, condC 2 | none | condA, condC 3 | condC | condA, condC 4 | condB | condB 5 | none | none
I was thinking to create a confusion matrix, by considering for example:
instance 1: 1 TP (condB), 1 FP (condA), 1 FN (condC) instance 2: 2 FN (condA, condC) instance 3: 1 TP (condC), 1 FN (condA) instance 4: 1 TP instance 5: 1 TN
and to derive precision, sensitivity and so on for the test. I have results for two separate tests predicting the same conditions, and then the results for a few cases when both of the tests have been used. So, I was planning to repeat the procedure for the 2 tests independently, and for the combinations of both, and comparing the derived precision, etc.
So, I have two questions:
- does it make sense to approach the problem this way? Is there another better way to evaluate the situation?
- For the 2 tests, I have different number of instances...is it still ok to compare their parameters (precision, recall) in order to establish which one is better?