In the context of record linkage, data de-duplication, or entity resolution, we attempt to merge entities that refer to the same thing into a single object. The obvious example is an address database where there are multiple entities for the same John Doe, but perhaps with slightly different spellings, mistakes or other variations in the record attributes.
We can solve this problem by training a model to learn the similarity/differences between pairs of records, i.e. for a given pair containing two records a and b, output a probability that both records are similar.
In order to use supervised learning for this problem, we need labels to train with. We can use a manual process to gather ground truth data; for example
ID | First | Last | cluster 1 | John | Doe | A 2 | Johnn | Doe | A 3 | Jon | Doe | A 4 | Jon | Snow | B 5 | John | Snow | B
We can generate pairs by grouping by the cluster for positive labels, then randomly grouping across clusters for negative labels. For the above example:
Generated positive labeled pairs: (1, 2) (1, 3) (2, 3) (4, 5) Generated negative labeled pairs: (1, 4) (1, 5) (2, 4) (2, 5) (3, 4) (3, 5)
Now for my question:
When creating train and test splits using all the generated pairs above, does a simple random split violate the assumptions of i.i.d.? There is a concern that splitting pairs generated from the same cluster (e.g. (1, 2), (1, 3) in train; (2, 3) in test) unfairly affects evaluation metrics because they were generated from the same cluster.