0
$\begingroup$

I'm developing a classifier system to detect objects of interest in images. I want to report a score, but I'm a bit lost as to what the most fair and informative number is.

Sensitivity and specificity seem to be commonly used, but while I have numbers for true positive, false positives and false negatives, I'm not sure how to properly calculate 'true negative'.

$\endgroup$
1
0
$\begingroup$

To answer your first question: neither sensitivity or specificity are that good measures on their own. You'd probably want a d-prime measure which considers both. Even if in your case you are primarily interested in a system that can correctly identify the presence of an object in a scene (e.g. correctly identifiying the presence of markers for cancer in an xray) rather than a system with high specificity (e.g. a systme that can correctly say there are no markers for cancer present) - d prime will still be a more useful measure, and practical importance of correctly spotting objects vs correctly spotting an absence of objects can be factored into this with weighting

$\endgroup$
0
$\begingroup$

You should be presenting your system with a number of scenes that either contain objects of interest or do not contain such objects (e.g. 10 of each). In this situation a 'true negative' is where your system correctly fails to respond when it is presented with a scene containing no objects of interest, a 'correct rejection'. The system could be set up to respond 'not containing objects' in this case -this would also be a correct rejection.

If you present 10 images without objects of interest and 10 containing objects of interest your classifier might identify:

8 scenes containing objects of interest as 'containing objects' (8 hits),
2 scenes containing objects of interest, as 'not containing objects' (2 misses),
4 scenes not containing objects of interest as 'containing objects of interest' (4 false alarms), 6 scenes not containing objects as 'not containing objects of interest' (6 correct rejections).

$\endgroup$
2
  • $\begingroup$ ok, but if I randomly choose regions they are very unlikely to contain either 0 or 1 object, most likely they will have several, some only partly in the image. It seems like I would need to curate a test set, and this seems artificial compared to the real use case where the system is given a large image and returns a list of objects and positions. $\endgroup$ – so12311 Mar 17 '14 at 14:19
  • $\begingroup$ Hi - sounds like you are interested in an 'unsupervised' classification procedure - i.e. you don't have a test set or 'training set'? Some more detail on your problem might be useful. $\endgroup$ – user41270 Mar 17 '14 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.