5
$\begingroup$

I'm working on merging records from several databases that cover the same entities, but share no reliably deterministic fields, leaving us with fields such as name and address to resolve identity. In reading about this problem I came across the Fellegi-Sunter statistical method for resolving record linkage.

I can't tell from my reading, however, exactly how the U probability should be determined. I know that it is the likelihood of two "randomly" paired records matching on a given field, but it's the "random" part that I'm struggling with. I'm working with a quantity of records that makes it impossible to compare all record pairings. For the actual comparison stage, I'll be "blocking" the records using zip code, but when generating U probabilities, this would seem to contradict the "randomly paired" idea.

Is it acceptable to compare records that fall in the same "block" for purposes of generating the U probabilities, or is there some other method, such as a sample of truly random pairings, that I should be using?

Improve this question
$\endgroup$
1
  • $\begingroup$ The $U$ probabilities should come from domain knowledge about the data itself. For example, if comparing birth month, the probability of two non-matching records having the same birth month is approximately $1/12$ (in theory). $\endgroup$
    – shabbychef
    Commented Jan 7, 2016 at 23:44

1 Answer 1

Reset to default
2
$\begingroup$

The u probabilities are relatively easy to estimate if we can assume that the overwhelming majority of comparisons are non-matches in the set of all comparisons. This assumption typically holds.

Where this is the case we can simply take a random sample of input rows and compute the cartesian product to generate all possible comparisons.

We then compute the U values by assuming that all these comparisons are non-matches.

We have an implementation of this approach in Splink, a piece of software which estimates the Fellegi Sunter model. The relevant code is here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.