Questions tagged [record-linkage]

Record linkage is the task of finding record(s) in a (or more) dataset(s) that refer to the same entity.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Estimating true and false matches from record linkage between two datasets

Suppose I have two datasets I want to join. Dataset A keeps track of users of a local health care service, identifying them by an unique key. Dataset B records all hospitalization events from citizens ...
0
votes
0answers
12 views

Planning exploratory factor analysis at the organisation level: is my data appropriate?

I am attempting to model (let us say) education system responses to student safety incidents, how schools learn from these incidents and what improves this process. My statistics is basic undergrad ...
1
vote
2answers
161 views

Confidence Intervals of the Positive Predictive Value: Adjusting my Bootstrap

I have two data sets, A and B, and I have linked them together using automatic, probabilistic methods to create a set of linked records (L). For example, my code could have predicted that record 1 in ...
0
votes
1answer
65 views

Referring multiple names to the same entity

I am working on the models of different product types and wish to generalize them to the same entity. For example, from the given list ...
1
vote
0answers
44 views

How to evaluate the performance of a predictor on an unlabelled dataset? What is an appropriate test set size and how to sample it?

I am working on a project with goal to deduplicate a customer database. We don't have any annotations (no training/test set with the ground truth values). We implemented multiple unsupervised ...
8
votes
1answer
4k views

Deep Learning approaches for Record Linkage

Record linkage (RL) is the task of finding records in a data set that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Record linkage is ...
2
votes
0answers
60 views

i.i.d. assumption for pairwise data generated from clusters

In the context of record linkage, data de-duplication, or entity resolution, we attempt to merge entities that refer to the same thing into a single object. The obvious example is an address database ...
3
votes
1answer
4k views

how can I train the model to predict duplicate records in machine learning?

I have the adjustment data in telecom domain: There are problems that sometimes the automation script generates the bill twice which causes duplicate records. I want to know how to prepare the data ...
3
votes
2answers
1k views

Learn a mapping between two datasets using Neural Network

I have two matrices $A_1$ of size $N\times K$ and $A_2$ of size $M\times K$ which contain data and every row has a corresponding label $y \in {1, 2, 3}$. I want to learn a mapping between those two ...
2
votes
1answer
42 views

Privacy preserved record linkage

What is the best way for privacy preserved record linkage in data mining. I am final year university student. My thesis topic is related to data mining. I am working with data and I need to privacy ...
1
vote
0answers
368 views

Machine learning matching methods?

My goal is match each of the invoices created to each payment made. So for example, if I have a set of invoices: ...
1
vote
1answer
87 views

Is there a valid way to merge sampled/panel data with exhaustive usage data?

We have an existing study of media consumption which is online and offline. Online study: To measure consumption online, we use log data from web accesses and various other sources. We have reliable ...
2
votes
2answers
222 views

Linkage of large datasets

Let me set the scene I have two data sources, one is 16e6 rows of data, the other is 160k rows of data. Each of these have text string identifiers [e.g. some allegory of 'name', 'address', 'postcode']....
8
votes
4answers
13k views

Popular named entity resolution software

I am working on a project and need to extract persons' names from a large amount of documents. This task should belong to the named entity resolution problem. What are currently some of the most ...
2
votes
1answer
367 views

Explanation of Algorithm within EpiLink

I'm working on doing some record linkage and have come across two choices for weighting and classifying pairs of entries. One being EpiLink proposed by P. Contiero et al. in "The EpiLink record ...
4
votes
0answers
174 views

Record linkage when sources have different fields

I have read a little about record linkage, but it seems to me that a requirement is that all fields in both sources can be compared. For example, with sources A and B, an assumption is that we can ...
13
votes
1answer
3k views

State-of-the-art in deduplication

What are the state-of-the-art methods in record deduplication? Deduplication is also sometimes called: record linkage, entity resolution, identity resolution, merge/purge. I know for example about ...
4
votes
1answer
351 views

Generating M/U Probabilities in Fellegi-Sunter Record Linkage

I'm working on merging records from several databases that cover the same entities, but share no reliably deterministic fields, leaving us with fields such as name and address to resolve identity. In ...
3
votes
0answers
97 views

Fuzzy Record Linkage of Spatial Datasets

I have two datasets describing real-estate properties Dataset 1 describes building characteristics; it includes the location of the entrance to the building along with building descriptions and ...
5
votes
1answer
371 views

Record linkage: Weighting matches by estimate of match quality

This question is somewhat of a continuation from an earlier posting: Using EM algorithm for record linking I have two data sets of individual some of whom are in both but a prior it is not know which ...
10
votes
3answers
3k views

Using EM algorithm for record linking

I am interested in linking records across 2 datasets by first name, last name, and birth year. Might this be doable with the EM algorithm, and if so, how? Consider the following record in the 1st as ...