Choosing the right threshold for a biometric trait authentication system

I have a biometric authentication system that is using a person's gait to authenticate them. I extract features from gait, run it through a comparison versus a template and produce a similarity score (where if this similarity score is below a certain threshold, then the user is authenticated). So, I have 72 trials total (36 trials containing a positive case and 36 that contain a negative case). What I want to do is graph the ability of this system to authenticate people by illustrating it with a ROC graph.

Unfortunately, I don't quite understand how to choose a threshold. Is there some mathematical procedure involved for choosing a threshold for the similarity scores? Do I just choose a bunch of different thresholds, and graph the corresponding ROC curves for all these different threshold values? The resulting similarity scores vary from [0.6,1.2] where the positive cases tend to lie around 0.6. All my coding is being done in Matlab.

• Have a look at the answers to this question on Stackoverflow to find the 'elbow' in the ROC. – Jonas Feb 7 '11 at 17:32

Generally, the cut-off value is chosen such as to maximize the compromise between sensitivity (Se) and specificity (Sp). You can generate a regular sequence of thresholds and plot the resulting ROC curve, as shown below, based on the DiagnosisMed R package.

Actually, the raw data looks like

  test.values TP FN FP TN Sensitivity Specificity
1       0.037 51  0 97  0           1      0.0000
2       0.038 51  0 96  1           1      0.0103
3       0.039 51  0 91  6           1      0.0619
4       0.040 51  0 84 13           1      0.1340
5       0.041 51  0 74 23           1      0.2371
6       0.042 51  0 67 30           1      0.3093


and the optimal threshold is found as

   test.values TP FN FP TN Sensitivity Specificity
47       0.194 43  8  8 89      0.8431      0.9175


To sum up, I would suggest to generate a regular sequence of possible thresholds and compute Se and Sp in each case; then, choose the one that maximize Se and (1-Sp) (or use other criteria if you want to minimize FP or FN rates).

• Shouldn't "optimality" take into account the cost of false positives versus false negatives? I can imagine a scenario where a false negative is far more costly than a false positive, which should call for a criterion biased towards capturing as much as the true distribution as possible. Also, I believe you mean "sensitivity" and not "sensibility" in your answer. – Mike Lawrence Dec 5 '10 at 13:22
• @Mike Yes, your comment is really welcome and this was what I tried to point to in my last sentence (optimize FN/FP rates depending on the context). Thanks for noticing the typo (which was already addressed in a preceding post...)! – chl Dec 5 '10 at 13:31
• So just to clarify, I have my data in the range of [0.6,1.2], then I calculate the corresponding ROC curve for a bunch of thresholds in that range. I find the cut-off points in each of those ROC curves and I try to pin-point the one that has the maximum Sensitivity/Specificity combination. Is that correct or have I completely misinterpreted you? – rohanbk Dec 5 '10 at 19:48
• @rohanbk Yes, that's basically the idea; e.g., compute Se/Sp and the four cells of your two-way table (TP, FN, etc.) in the range [0.6,1.2] (say with an increment of .05 or less -- it depends on the precision you can/want to attain in the future). Choose the compromise Se/(1-Sp) that best reflects your classification objective. Some references were provided to you in earlier threads. I don't really know if there are good MATLAB functions for ROC analysis, but I just found this page, j.mp/ekzg3m. HTH – chl Dec 5 '10 at 20:23