3
$\begingroup$

I have the adjustment data in telecom domain: There are problems that sometimes the automation script generates the bill twice which causes duplicate records.

I want to know how to prepare the data and train the ML model. Currently, I am considering the duplicate record as issues and training the SVM model for one-class classification.

Duplicate Bill
    ABC13C,13.59,2017-03-01,2017-03-01,2017-03-01,118,I,A,0,C,2011    1

Should I train the model with actual bill and duplicate bill as both issues and train the model and then predict the rest or is there any other way to consider or should I implement time-series classification?

Records 
Actual Bill                                                     label
ABC13C,13.59,2017-03-01,2017-03-01,2017-03-01,118,I,A,0,C,2011    1
Duplicate Bill
ABC13C,13.59,2017-03-01,2017-03-01,2017-03-01,118,I,A,0,C,2011    1
Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Why don't you just check for duplicates with DISTINCT in sql, or just compare the id's or something? What should your algoritm predict, based on what? $\endgroup$
    – Gijs
    Commented Oct 23, 2017 at 20:58
  • $\begingroup$ I have millions of records to check some time amount will be different $\endgroup$
    – vinaykva
    Commented Oct 23, 2017 at 21:01
  • $\begingroup$ sometime they will be legitimate transaction which is being adjusted for the previous months $\endgroup$
    – vinaykva
    Commented Oct 23, 2017 at 21:04

1 Answer 1

Reset to default
2
$\begingroup$

There are packages available in R and Python which can help you with de-duplicating.

Python record linkage -> http://recordlinkage.readthedocs.io/en/latest/about.html

R record linkage -> https://cran.r-project.org/web/packages/RecordLinkage/index.html

Also refer to Deep Learning approaches for Record Linkage

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.