2
$\begingroup$

I have a large data set (17.000 cases) with 9.000 distinct company names. However, most of the names are duplicates/misspellings.

Is there a way to identify the duplicates/misspellings by an algorithm?

At the moment I use grepl to identify the duplicates. Like:

dat %>%
select(Company) %>%
filter(grepl("g&s|g & s", ignore.case = T, Company)) %>%
distinct()

# A tibble: 15 x 1
 Company                                                  
   <chr>                                                    
 1 das projekt Projektmanagement, Consulting & Services GmbH
 2 G&S Vastgoed                                             
 3 G&S Bouw                                                                                      
 5 G&S                                                      
 6 G&S Hotelbetriebs GmbH                                   
 7 BCadvies - G&S                                           
 8 G&s Bouw                                                 
 9 G&S bouw

After that, I standardize the names like:

dat = dat %>%
  mutate(Company = ifelse(grepl("g&s|g & s", ignore.case = T, Company), "G & S", Company))
Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ There are many! Some relevant terms of art include "entity resolution" and "record linkage." $\endgroup$
    – Sycorax ♦
    Feb 13, 2019 at 15:26

1 Answer 1

Reset to default
1
$\begingroup$

when I had to deal with that problem in R I started here https://www.r-bloggers.com/fuzzy-string-matching-a-survival-skill-to-tackle-unstructured-information/

Currently I am using the fuzzywuzzyR https://cran.r-project.org/web/packages/fuzzywuzzyR/index.html

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.