Best ML technique to identify customer records that were duplicated within the system

Question

I am looking, as a theoretical exercise, which algorithm/ technique could be the best one to identify customer records that were duplicated within a retail company system. For example, when a company has someone’s name, mailing address or email entered differently from one shopping experience to the next.”

For example, a shopper is in the system as “Christine” in one record and “Chris” in another. Or an address is “123 Main Street, Apartment B” in one record, but “123-B Main Street” in another. How could be a good ML technique to determine when multiple records actually represent the same person and link them together.

Answer 1

Different fields may need different logic. Let's take an address as an example. Even if address is represented as 2 different strings, geocoding services (e.g. Google Maps) should still resolve them to the same longitude and latitude. Matching 2 numbers should be trivial to match between records.

Names usually have a canonical form (or several, e.g. "Alex" may be just "Alex" or short version of "Alexander" or "Alexey" in some countries).

For more complicated cases such as matching profession, profile description, etc. you can train a siamese network which is specifically designed to distinguish between 2 views of the same object and 2 different objects (e.g. someone's face of profile description). The downside, of course, is that you will need to collect a dataset for training.