I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm thinking

  • Maybe some kind of transformer neural network without positional encoding (there's not really an obvious ordering, so LSTM or transformer with positional encoding seem less obvious). The number of records per genre in the real data is low enough that sequence length ought to be okay.
  • Additionally, using pre-trained encoder for language may seem really obvious for capturing embeddings for the text fields, to capture information models have seen during training of role/film/actor name.
  • While the text information will in fact often manage to directly produce the match, working with text only may often not be enough, because we also need to use the date information that may often matter (esp. when things are ambigious/incomplete).
  • Possibly multiple binary losses?
  • Many other models (e.g. GBDTs etc.) cannot deal with the multiple-inputs-multiple-outputs format (while they could of course take in text embeddings).

Below are examples of what the data might look like (I cannot share the real data, but the below shares the core features):

Genre show_movie year_start year_end
Science Fiction Star Trek (original series) 1966 1969
Science Fiction Star Trek: The Motion Picture 1979 1979
Science Fiction Star Trek (2009) 2009 2009
Science Fiction Star Trek: Strange New Worlds 2022
Superhero Man of Steel 2013 2013
Superhero Superman 1948 1948
Superhero Batman v Superman: Dawn of Justice 2016 2016
Superhero Daredevil 2003 2003
Action Mission: Impossible 1966 1973
Action The Great Train Robbery 1903 1903


genre role actor born died
Science Fiction Spock Leonard Nimoy 1931 2015
Action Paris Leonard Nimoy 1931 2015
Science Fiction Spock Quinto, Zachary 1977
Science Fiction Spock Ethan Peck 1986
Superhero Superman Cavill, Henry 1983
Superhero Matt Murdock, Daredevil and Batman Ben Affleck 1972
Superhero A superhero with a red cape Kirk Alyn 1910 1999
Action J. D. B. 1862 1946

The goal is to obtain this:

genre show_movie role actor
Science Fiction Star Trek (original series) Spock Leonard Nimoy
Science Fiction Star Trek: The Motion Picture Spock Leonard Nimoy
Science Fiction Star Trek (2009) Spock Quinto, Zachary
Science Fiction Star Trek: Strange New Worlds Spock Ethan Peck
Superhero Superman A superhero with a red cape Kirk Alyn
Superhero Man of Steel Superman Cavill, Henry
Superhero Batman v Superman: Dawn of Justice Superman Cavill, Henry
Superhero Badman vs supreman - dawn of justice Batman Ben Affleck
Superhero Daredevil Daredevil Ben Affleck
Superhero Daredevil Matt Murdoch Ben Affleck
Action Mission: Impossible Paris Leonard Nimoy
Action The Great Train Robbery J. D. B.

Key features of this are

  • We always have the genre information and only ever need to match within genre, but each genre has a different number of records that you need to match up.
  • Every entry in the first dataset has one or more matching entries in the second (but never 0)
  • Every entry in the second database has one or more matching entries in the first (but never 0)
  • Dates help for some extent, e.g. only Leonard Nimoy or Kirk Alyn could be actors in "Star Trek (original series)", because all others were born before it started. But both could not have been in "Batman v Superman: Dawn of Justice" because he died a year before (erm, okay so maybe that is not so clear, because Nimoy would only have had to be alive during filming). In contrast, Ben Affleck or Zachary Quinto could have been in Star Trek: The Motion Picture at ages 7 and 2 respectively.
  • There's typos and sometimes vaguer/more general terms that you need context knowledge to match up (e.g. "Superman" is a "Superhero with a red cape", but other superheros could qualify). There's also no perfect adherence to any formatting (e.g. multiple entries separated by ",", ";", "/", "and", or longer text fragments.
  • It's not always clear whether it will be possible to perfectly match things up if some information is too generic (e.g. if we had an entry for "Female main character", "Scarlett Johansson").

1 Answer 1


Sounds like an interesting problem! I would approach this with a mixed strategy, rather than a single ML model to do the matching. You have the right intuition here about using a pre-trained text encoder. Have you tried this idea out first?

For example, you can encode show_movie in table 1 and role in table 2 into embeddings, and then use a vector database (Pinecone and Chroma to name two) to do efficient similarity matching for these columns. With this, you can match rows if they pass a minimum similarity (such as 0.9). The likelihood of this similarity doing well will depend on the data the text encoder was trained on -- you can look into models that are trained or finetuned on databases like IMDB, or finetune one yourself! A BERT encoder should work fine.

If you observe the precision isn't as high as you'd like (i.e. too many matches), you can further filter using standard SQL on the dates:

WHERE born < year_start AND year_end < died

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.