Questions tagged [record-linkage]
Record linkage is the task of finding record(s) in a (or more) dataset(s) that refer to the same entity.
21 questions
1
vote
1
answer
215
views
Machine learning model for matching records
I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm ...
- 35.2k
0
votes
0
answers
29
views
Modeling Temporal-Interval Distance
I'm trying to define a model for comparing temporal intervals, and can't find an appropriate distance function that incorporates the different relations two temporal intervals might have, e.g. overlap,...
1
vote
2
answers
599
views
Confidence Intervals of the Positive Predictive Value: Adjusting my Bootstrap
I have two data sets, A and B, and I have linked them together using automatic, probabilistic methods to create a set of linked records (L). For example, my code could have predicted that record 1 in ...
- 179
1
vote
1
answer
173
views
Referring multiple names to the same entity
I am working on the models of different product types and wish to generalize them to the same entity. For example, from the given list
...
- 113
1
vote
0
answers
47
views
How to evaluate the performance of a predictor on an unlabelled dataset? What is an appropriate test set size and how to sample it?
I am working on a project with goal to deduplicate a customer database. We don't have any annotations (no training/test set with the ground truth values).
We implemented multiple unsupervised ...
- 11
8
votes
2
answers
5k
views
Deep Learning approaches for Record Linkage
Record linkage (RL) is the task of finding records in a data set that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Record linkage is ...
- 201
2
votes
0
answers
75
views
i.i.d. assumption for pairwise data generated from clusters
In the context of record linkage, data de-duplication, or entity resolution, we attempt to merge entities that refer to the same thing into a single object. The obvious example is an address database ...
- 21
3
votes
1
answer
6k
views
how can I train the model to predict duplicate records in machine learning?
I have the adjustment data in telecom domain:
There are problems that sometimes the automation script generates the bill twice which causes duplicate records.
I want to know how to prepare the data ...
- 409
4
votes
2
answers
3k
views
Learn a mapping between two datasets using Neural Network
I have two matrices $A_1$ of size $N\times K$ and $A_2$ of size $M\times K$ which contain data and every row has a corresponding label $y \in {1, 2, 3}$. I want to learn a mapping between those two ...
- 61
2
votes
1
answer
49
views
Privacy preserved record linkage
What is the best way for privacy preserved record linkage in data mining. I am final year university student. My thesis topic is related to data mining. I am working with data and I need to privacy ...
- 21
1
vote
0
answers
420
views
Machine learning matching methods?
My goal is match each of the invoices created to each payment made. So for example, if I have a set of invoices:
...
- 111
1
vote
1
answer
185
views
Is there a valid way to merge sampled/panel data with exhaustive usage data?
We have an existing study of media consumption which is online and offline.
Online study: To measure consumption online, we use log data from web accesses and various other sources. We have reliable ...
- 13
2
votes
2
answers
582
views
Linkage of large datasets
Let me set the scene
I have two data sources, one is 16e6 rows of data, the other is 160k rows of data. Each of these have text string identifiers [e.g. some allegory of 'name', 'address', 'postcode']....
- 23
11
votes
5
answers
14k
views
Popular named entity resolution software
I am working on a project and need to extract persons' names from a large amount of documents. This task should belong to the named entity resolution problem. What are currently some of the most ...
2
votes
1
answer
401
views
Explanation of Algorithm within EpiLink
I'm working on doing some record linkage and have come across two choices for weighting and classifying pairs of entries. One being EpiLink proposed by P. Contiero et al. in "The EpiLink record ...
- 33
4
votes
0
answers
191
views
Record linkage when sources have different fields
I have read a little about record linkage, but it seems to me that a requirement is that all fields in both sources can be compared.
For example, with sources A and B, an assumption is that we can ...
- 151
14
votes
2
answers
3k
views
State-of-the-art in deduplication
What are the state-of-the-art methods in record deduplication? Deduplication is also sometimes called: record linkage, entity resolution, identity resolution, merge/purge. I know for example about ...
- 241
5
votes
1
answer
631
views
Generating M/U Probabilities in Fellegi-Sunter Record Linkage
I'm working on merging records from several databases that cover the same entities, but share no reliably deterministic fields, leaving us with fields such as name and address to resolve identity. In ...
- 151
3
votes
0
answers
116
views
Fuzzy Record Linkage of Spatial Datasets
I have two datasets describing real-estate properties
Dataset 1 describes building characteristics; it includes the location of the entrance to the building along with building descriptions and ...
- 145
5
votes
1
answer
438
views
Record linkage: Weighting matches by estimate of match quality
This question is somewhat of a continuation from an earlier posting:
Using EM algorithm for record linking
I have two data sets of individual some of whom are in both but a prior it is not know which ...
- 815
10
votes
3
answers
3k
views
Using EM algorithm for record linking
I am interested in linking records across 2 datasets by first name, last name, and birth year. Might this be doable with the EM algorithm, and if so, how?
Consider the following record in the 1st as ...
- 815