5
$\begingroup$

I have a biometric authentication system that is using a person's gait to authenticate them. I extract features from gait, run it through a comparison versus a template and produce a similarity score (where if this similarity score is below a certain threshold, then the user is authenticated). So, I have 72 trials total (36 trials containing a positive case and 36 that contain a negative case). What I want to do is graph the ability of this system to authenticate people by illustrating it with a ROC graph.

Unfortunately, I don't quite understand how to choose a threshold. Is there some mathematical procedure involved for choosing a threshold for the similarity scores? Do I just choose a bunch of different thresholds, and graph the corresponding ROC curves for all these different threshold values? The resulting similarity scores vary from [0.6,1.2] where the positive cases tend to lie around 0.6. All my coding is being done in Matlab.

$\endgroup$
  • $\begingroup$ Have a look at the answers to this question on Stackoverflow to find the 'elbow' in the ROC. $\endgroup$ – Jonas Feb 7 '11 at 17:32
4
$\begingroup$

Generally, the cut-off value is chosen such as to maximize the compromise between sensitivity (Se) and specificity (Sp). You can generate a regular sequence of thresholds and plot the resulting ROC curve, as shown below, based on the DiagnosisMed R package.

alt text

Actually, the raw data looks like

  test.values TP FN FP TN Sensitivity Specificity
1       0.037 51  0 97  0           1      0.0000
2       0.038 51  0 96  1           1      0.0103
3       0.039 51  0 91  6           1      0.0619
4       0.040 51  0 84 13           1      0.1340
5       0.041 51  0 74 23           1      0.2371
6       0.042 51  0 67 30           1      0.3093

and the optimal threshold is found as

   test.values TP FN FP TN Sensitivity Specificity
47       0.194 43  8  8 89      0.8431      0.9175

To sum up, I would suggest to generate a regular sequence of possible thresholds and compute Se and Sp in each case; then, choose the one that maximize Se and (1-Sp) (or use other criteria if you want to minimize FP or FN rates).

$\endgroup$
  • $\begingroup$ Shouldn't "optimality" take into account the cost of false positives versus false negatives? I can imagine a scenario where a false negative is far more costly than a false positive, which should call for a criterion biased towards capturing as much as the true distribution as possible. Also, I believe you mean "sensitivity" and not "sensibility" in your answer. $\endgroup$ – Mike Lawrence Dec 5 '10 at 13:22
  • $\begingroup$ @Mike Yes, your comment is really welcome and this was what I tried to point to in my last sentence (optimize FN/FP rates depending on the context). Thanks for noticing the typo (which was already addressed in a preceding post...)! $\endgroup$ – chl Dec 5 '10 at 13:31
  • $\begingroup$ So just to clarify, I have my data in the range of [0.6,1.2], then I calculate the corresponding ROC curve for a bunch of thresholds in that range. I find the cut-off points in each of those ROC curves and I try to pin-point the one that has the maximum Sensitivity/Specificity combination. Is that correct or have I completely misinterpreted you? $\endgroup$ – rohanbk Dec 5 '10 at 19:48
  • $\begingroup$ @rohanbk Yes, that's basically the idea; e.g., compute Se/Sp and the four cells of your two-way table (TP, FN, etc.) in the range [0.6,1.2] (say with an increment of .05 or less -- it depends on the precision you can/want to attain in the future). Choose the compromise Se/(1-Sp) that best reflects your classification objective. Some references were provided to you in earlier threads. I don't really know if there are good MATLAB functions for ROC analysis, but I just found this page, j.mp/ekzg3m. HTH $\endgroup$ – chl Dec 5 '10 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.