0
$\begingroup$

A reproducible example with a small bit of R code is available in this stackoverflow post (link so I dont need to re-type out the code). The fuzzytext library in R has the following available string methods c("osa", "lv", "dl", "hamming", "lcs", "qgram", "cosine", "jaccard", "jw", "soundex"). Our use case is matching (left-joining) basketball player names from 2 different sources. From the stackoverflow post, we have the following concerns to account for when string matching names:

  • The left join shouldn't get mixed up by long / short names. Michael Gadson is clearly Mike Gadson, not one of the other Mike names in the dataset with a different last name.
  • The left join shouldn't get mixed up by reversed names. Ricky Smith is Rick Smith, he is not Smith Rickie.
  • The left join shouldn't get mixed up by III, Jr., etc. suffix to names, or by extra spaces or symbols: eg. De Andre' vs DeAndre)
  • Certain players (e.g. Johnny Williams) in the left-hand-side dataframe have no match in the right-side table. To catch this, we'll need to rely on a properly selected max_dist value.

A 5th concern is avoiding duplicates in the code (we want only 1 row for each person in the left-hand-side dataframe), however this is handled with the groupby(fullName) %>% filter(dist == min(dist) | is.na(dist)) in the code.

Our question is then: given these concerns, what is a good method and max distance to use for this left join?

$\endgroup$
1
$\begingroup$

The first point about long / short names is slightly difficult to do with edit distance alone. A two stage profess may work: use edit distances with a low maximum edit distance (to minimize false positives) first and then manually fix any remaining non-matches.

I would try the Damerau-Levenshtein or maybe the osa. The Wikipedia edit distance page gives a good summary.

From the stringdist package documentation:

The full Damerau-Levenshtein distance (method='dl') is like the optimal string alignment distance except that it allows for multiple edits on substrings.

The Optimal String Alignment distance (method='osa') is like the Levenshtein distance but also allows transposition of adjacent characters. Here, each substring may be edited only once. (For example, a character cannot be transposed twice to move it forward in the string).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.