5
$\begingroup$

I have a biometric system that outputs a distribution of scores that resembles a Gaussian distribution (similar to the example graph in the following link: LINK). My point of confusion is how I calculate the False Acceptance Rate. How does threshold factor into the whole problem?

$\endgroup$
12
$\begingroup$

Just to add to other responses, here is a brief recap' on terminology.

For any biometric or classification system, the main performance indicator is the receiver operating characteristic (ROC) curve, which is a plot of true acceptance rate (TAR=1-FRR, the false rejection rate) against false acceptance rate (FAR), which is computed as the number of false instances classified as positive among all intruder and impostor cases. The closer the curve is to the top left corner, the better it is (this corresponds to maximizing the so-called area under the curve or AUC). Generally, such curves are generated offline from a database of previous records. In the biometric literature, FAR is sometimes defined such that the "impostor" makes zero effort to obtain a match. Here, I'm roughly quoting Biometrics, from Boulgouris et al. (chap. 26).

So, you may choose your cutoff by using standard ROC tools (search for "ROC analysis" on Rseek) to find the best compromise between FAR and TAR (this is not necessarily that cutoff that maximizes the AUC, it depends on your objectives).

Now, as has been highlighted in other responses, this compromise between FAR and TAR led to similar interpretation in psychophysics, classification, or biomedical science. It's just a matter of terminology, and we often speak of Hit rate vs. False Alarm rate; sensibility vs. specificty.

Note

Here are some pictures to complement other responses, which I hope will help you to draw the parallel with decision theory and statistical testing.

Let an individual be facing a two-alternative choice experiment. Depending on the location of his internal criterion, his response may lead to Hit or False Alarm (response > criterion), or alternatively Correct Rejection or Miss (response < criterion). The corresponding probabilistic response curve resemble your situation.

alt text

Most classical textbooks on Statistics provide a Table similar to the one below, where we describe the probabilities of incorrectly rejecting a null hypothesis ($\alpha$) vs. falsely “accepting” the null ($\beta$) where in fact the alternative is true.

alt text

This leads to quite the same picture as with the psychophysical threshold model: alt text

$\endgroup$
  • 1
    $\begingroup$ The enclosed pictures were my first attempt at using Asymptote (j.mp/c8XUGq) instead of Metapost :-) Very sad idea, but I can share the code if you like. $\endgroup$ – chl Oct 16 '10 at 9:10
  • $\begingroup$ That's impressive; please share. :) $\endgroup$ – ars Oct 16 '10 at 10:09
  • 3
    $\begingroup$ @ars Here it is (as Gist): gist.github.com/629642, gist.github.com/629644, gist.github.com/629645. $\endgroup$ – chl Oct 16 '10 at 10:20
5
$\begingroup$

I'm not certain. I'm curious as to the other responses you get. However, I think you'll need to clarify a bit:

Does your Gaussian distribution represent the scores for a population of individuals which should be rejected by your biometric system?

If so, then I think you simply need to compute a cumulative probability - i.e. the percentage of individuals which should be rejected but who, by random chance, fall above your threshold and are "falsely accepted" by your biometric device.

So, it could be as simply as computing the number of people who randomly fall above your threshold divided by the total number of "should be rejected" people.

But again, I'm not certain of my response and I think you need to clarify what your assumptions are, what your threshold is, and how you wish to classify individuals as "falsely rejected".

$\endgroup$
  • $\begingroup$ Thanks for your answer. Yes, the distribution does represent the scores for the population that should be rejection by my biometric system but are instead authenticated. However, I'm not sure how to choose my threshold in this case, so I need to determine that before I can proceed. $\endgroup$ – rohanbk Oct 11 '10 at 18:04
  • 2
    $\begingroup$ I'd bet that the threshold would be a design parameter set by the client. Basically it will come down to which is more important for a specific client Type I or Type II error (probably Type II - Falsely accepting someone who should've been denied access). With a knowledge of what the client wants, you could then state that under a specific threshold, the probability of falsely admitting someone who should be denied access is 1 / 10,000,000 or something. $\endgroup$ – M. Tibbits Oct 11 '10 at 18:09
  • $\begingroup$ I could however envision a client who must employ a certain biometric authentication device due to government law or otherwise -- and said individuals might hate when their scanner goes wonky and won't give anyone access -- hence, they might care more about Type I error - not admitting someone who should be allowed (because they probably have six other security measures and this one is only a deterrent for wandering eyes... $\endgroup$ – M. Tibbits Oct 11 '10 at 18:12
  • $\begingroup$ So, does the threshold need to be the minimum score that needs to be obtained in order to be authenticated? $\endgroup$ – rohanbk Oct 11 '10 at 18:53
  • 1
    $\begingroup$ @rohanbk Did you look at Signal Detection Theory or ROC curve analysis (j.mp/b49wDl in French, j.mp/aTjobH in English)? Determining the cut-off is exactly finding the best compromise between sensitivity and 1-specificity, as M. Tibbits said. $\endgroup$ – chl Oct 11 '10 at 21:23
5
$\begingroup$

it sounds as tho the following simplified situation may capture the essence of your problem:

there are two populations of individuals: A = acceptable individuals and U = unacceptables. associated with each individual is a 'score' $X$. suppose in each of the two populations, the scores have gaussian distributions, where in A, the [true] mean is $\mu_A$ and in U, it is $\mu_U$. we can also suppose [altho need not] that the distributions have the same SD = $\sigma$. all three [or four] parameters are presumably known.

suppose $\mu_A > \mu_U$, so it makes sense to accept an individual if their 'score' $X$ is above some threshold $c$, say.

there are two ways this rule can go wrong:

  1. an $X$ from U can exceed $c$, leading to a false acceptance.

  2. an $X$ from A can be below $c$, leading to a false rejection.

the probabilities

$$err_{falseacc} = P(N(\mu_U, \sigma^2) > c)$$

and

$$err_{falserej} = P(N(\mu_A, \sigma^2) < c)$$

are the two error rates associated with the rule. you are focusing on $err_{falseacc}$.

it is not difficult to see that as the threshold $c$ is changed, one error-rate will decrease and the other will increase. so $c$ has to be chosen to give values of both error-rates that one can live with.

once you choose $c$, as others have already remarked, the error rates can be calculated.

in the language of statistics, you are testing two hypotheses about the $\mu$ of the population that the individual with observed score $X$ came from. one hypothesis is H$_A: \mu = \mu_A$ and the other is H$_U: \mu = \mu_U$. the 'test' to decide between these two hypotheses is the above rule and the error rates given above are [somewhat unhelpfully] called the type I and type II errors, or [equally unhelpfully, IMHO] the sensitivity and the specificity or [likewise] the producer's risk and the consumer's risk. which is which depends on which of the two hypotheses is designated as the 'null hypothesis', a distinction that may not be entirely helpful in this context.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.