Questions tagged [record-linkage]

Record linkage is the task of finding record(s) in a (or more) dataset(s) that refer to the same entity.

Filter by
Sorted by
Tagged with
0 votes
0 answers
27 views

Modeling Temporal-Interval Distance

I'm trying to define a model for comparing temporal intervals, and can't find an appropriate distance function that incorporates the different relations two temporal intervals might have, e.g. overlap,...
user avatar
1 vote
2 answers
312 views

Confidence Intervals of the Positive Predictive Value: Adjusting my Bootstrap

I have two data sets, A and B, and I have linked them together using automatic, probabilistic methods to create a set of linked records (L). For example, my code could have predicted that record 1 in ...
user avatar
  • 179
1 vote
1 answer
120 views

Referring multiple names to the same entity

I am working on the models of different product types and wish to generalize them to the same entity. For example, from the given list ...
user avatar
  • 113
1 vote
0 answers
47 views

How to evaluate the performance of a predictor on an unlabelled dataset? What is an appropriate test set size and how to sample it?

I am working on a project with goal to deduplicate a customer database. We don't have any annotations (no training/test set with the ground truth values). We implemented multiple unsupervised ...
user avatar
8 votes
1 answer
4k views

Deep Learning approaches for Record Linkage

Record linkage (RL) is the task of finding records in a data set that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Record linkage is ...
user avatar
  • 201
2 votes
0 answers
73 views

i.i.d. assumption for pairwise data generated from clusters

In the context of record linkage, data de-duplication, or entity resolution, we attempt to merge entities that refer to the same thing into a single object. The obvious example is an address database ...
user avatar
  • 21
3 votes
1 answer
5k views

how can I train the model to predict duplicate records in machine learning?

I have the adjustment data in telecom domain: There are problems that sometimes the automation script generates the bill twice which causes duplicate records. I want to know how to prepare the data ...
user avatar
  • 389
3 votes
2 answers
2k views

Learn a mapping between two datasets using Neural Network

I have two matrices $A_1$ of size $N\times K$ and $A_2$ of size $M\times K$ which contain data and every row has a corresponding label $y \in {1, 2, 3}$. I want to learn a mapping between those two ...
user avatar
2 votes
1 answer
45 views

Privacy preserved record linkage

What is the best way for privacy preserved record linkage in data mining. I am final year university student. My thesis topic is related to data mining. I am working with data and I need to privacy ...
user avatar
1 vote
0 answers
377 views

Machine learning matching methods?

My goal is match each of the invoices created to each payment made. So for example, if I have a set of invoices: ...
user avatar
1 vote
1 answer
98 views

Is there a valid way to merge sampled/panel data with exhaustive usage data?

We have an existing study of media consumption which is online and offline. Online study: To measure consumption online, we use log data from web accesses and various other sources. We have reliable ...
user avatar
  • 13
2 votes
2 answers
318 views

Linkage of large datasets

Let me set the scene I have two data sources, one is 16e6 rows of data, the other is 160k rows of data. Each of these have text string identifiers [e.g. some allegory of 'name', 'address', 'postcode']....
user avatar
  • 23
11 votes
4 answers
13k views

Popular named entity resolution software

I am working on a project and need to extract persons' names from a large amount of documents. This task should belong to the named entity resolution problem. What are currently some of the most ...
user avatar
  • 1,147
2 votes
1 answer
374 views

Explanation of Algorithm within EpiLink

I'm working on doing some record linkage and have come across two choices for weighting and classifying pairs of entries. One being EpiLink proposed by P. Contiero et al. in "The EpiLink record ...
user avatar
4 votes
0 answers
178 views

Record linkage when sources have different fields

I have read a little about record linkage, but it seems to me that a requirement is that all fields in both sources can be compared. For example, with sources A and B, an assumption is that we can ...
user avatar
  • 141
13 votes
1 answer
3k views

State-of-the-art in deduplication

What are the state-of-the-art methods in record deduplication? Deduplication is also sometimes called: record linkage, entity resolution, identity resolution, merge/purge. I know for example about ...
user avatar
4 votes
1 answer
422 views

Generating M/U Probabilities in Fellegi-Sunter Record Linkage

I'm working on merging records from several databases that cover the same entities, but share no reliably deterministic fields, leaving us with fields such as name and address to resolve identity. In ...
user avatar
  • 141
3 votes
0 answers
102 views

Fuzzy Record Linkage of Spatial Datasets

I have two datasets describing real-estate properties Dataset 1 describes building characteristics; it includes the location of the entrance to the building along with building descriptions and ...
user avatar
  • 135
5 votes
1 answer
397 views

Record linkage: Weighting matches by estimate of match quality

This question is somewhat of a continuation from an earlier posting: Using EM algorithm for record linking I have two data sets of individual some of whom are in both but a prior it is not know which ...
user avatar
10 votes
3 answers
3k views

Using EM algorithm for record linking

I am interested in linking records across 2 datasets by first name, last name, and birth year. Might this be doable with the EM algorithm, and if so, how? Consider the following record in the 1st as ...
user avatar