Score-level fusion in biometrics

Question

I am looking into score-level fusion in biometrics. Even though I have read quite a lot of papers concerning this subject, I still can't wrap my head around one thing, and that is how the scores of the separate biometrics are combined (which is quite foundational to the fusion subject).

Imagine the situation where I want to fuse face and finger. Then I can assume one of the following:

• I assume that each sample pair is from the same user (subject1_finger_sample & subject1_face_sample)
• I assume that each sample pair can be a fraud as well (subject1_finger_sample & subject2_face_sample)

It seems most papers that I read assume the first, although they do not explain why. An example is given here. This means the following would be a valid test for the fusion algorithm:

(subject1_finger_sample + subject1_face_sample)
compared to
(subject2_finger_sample + subject2_face_sample)

But the following would not be:

(subject1_finger_sample + subject2_face_sample)
compared to
(subject2_finger_sample + subject1_face_sample)

Because they seem to assume that BOTH samples in a two-modal biometric setup will always be from the same person. However, in real life this seems counterintuitive: two impostors can easily work together, the first tries to forge my fingerprint, and the second moves his face in front of the webcam..

Any scientific reasoning for this?

Answer 1

For the fused score, there will be a decision boundary $y=f(x)$, where $x$ is the face score (say) and $y$ the fingerprint score, such that the subject is declared genuine iff $y\ge f(x)$.

Let's say that the scores are normalised to $[0,1]$ such that a fraction $1-x$ of random imposters have score bigger than $x$. The probability that a team of two independent random imposters can beat the system is then $$ 1\ -\ \int_0^1\!f(x)dx $$

For one random imposter, the probability depends on whether random people's face and fingerprint scores (relative to the genuine subject) are correlated or not. I'm unfamiliar with the literature, but I suspect there's a working assumption of uncorrelated. If, in fact, face and fingerprint scores are positively correlated (which seems just about plausible) then the formula above underestimates the single random imposter's chances.

So, mathematically speaking, one random imposter probably has the same or (perhaps slightly) better chance than two random imposters of breaking the system.

But are "random" impersonations really the right thing to consider? It might be practical, with a small dose of luck (possession of a good photograph) and a reasonable amount of work, for a criminal mastermind to find a near-match before attacking any single-factor system (e.g. face recogniser), raising his chances by a factor of 100, say. So the practical probability of impersonating a single-factor system may be 100 (say) times the random probability.

In a two-factor system, the criminal mastermind could (a) search for a single reasonably-close-matching individual and, as before, hope for 100 times impersonation gain over random; or (b) split his search for near face- and fingerprint-matches into two teams, each achieving a factor of 50 improvement on random, for a practical-vs-random gain of 2500 overall. So, yes, the practical impersonation probability does indeed depend on whether you attack the system with one or two imposters.

In summary, your two scenarios (one attacker or two) are probably so close mathematically in the random case that it makes no difference. I don't know the literature, but I suspect it's random-case figures that get quoted. On the other hand, there is a very large difference in the practical chances of a gang with sufficient collateral (grainy photos, mucky fingerprints etc) in the two scenarios, and perhaps the literature could do with being clearer on that subject.

Answer 2

I think you are concerned about the probability of a successful imposter attack? In this case imagine the imposter chooses subject x face and subject y fingerprints to try to gain unauthorised access to a system. Now imagine there is a list of authorised users 1 .. n. The authorisation system has to decide:

(i) Which user is closest (in feature space) to the information provided? (ii) Is the information provided sufficiently close to this user to grant access?

To answer the first question, one approach is to compare the information provided against the data of all authorised users. So we want to do the following comparisons:

(face x, fingerprints y) vs (face 1, fingerprints 1)

(face x, fingerprints y) vs (face 2, fingerprints 2)

. . .

(face x, fingerprints y) vs (face n, fingerprints n)

These comparisons could be used to a give a score. These scores would then provide the basis for answering question 1. Then a decision criteria (the simplest would be a threshold), can be applied to answer question 2.

So, a "good" system will apply very low scores to (face x, fingerprints y) if $x \neq y$. An attacker should want to choose x and y to be equal in order to gain access. If the system is good then choosing x and y to be from different subjects should never grant access.

For the comparison you give I think the process you are interested in is probably:

(face query data, fingerprint query data) vs (stored face 1, stored fingerprints 1) = "score 1"

(face query data, fingerprint query data) vs (stored face 2, stored fingerprints 2) = "score 2"

Compare score 1 and score 2 to decide which user is attempting to log in, and if the query data is sufficiently close to the stored information to give access (i.e. questions 1&2 above)

The comparison:

score = (stored face 1, stored fingerprints 1) vs (stored face 2, stored fingerprints 2)

Would be less common, but might be used, for example, to test how similar the two stored profiles are - i.e. how easy it is to tell them apart. That information might be used to improve the system.

In summary you can do any comparisons you want, but which one you should do depends on what question you are trying to answer.