5
$\begingroup$

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names.

For example,

"Shanghai, China" "China, ShangHai" might mean the same city.

Note this is slightly different (should be easier) than named entity recognition since we know in this field it's just city names/places.

My question is, how do I normalize these free text and match them with known city names in our database? We have a standard database in the format of

CityName, Country.

The best I can think of is sort of fuzzy string matching. Are there any existing research/tools doing this?

$\endgroup$
2
$\begingroup$

This task is typically referred as named entity normalization. Fuzzy string matching can be a good baseline if words are not too close (in terms of Levenshtein distance) in your dictionary. I have used the Python package fuzzywuzzy in the past for that purpose.

$\endgroup$
0
$\begingroup$

What I currently do to normalize places name is resorting to geocoding using api such as, e.g. navitia or google map, which already deal with the normalization process.

Once the lat/lng in hand, I reverse-geocode them, of course always using the same api so as to get normalized outputs.

Furthermore, these api return more information than just a normalized city name, but also information which allow to add line to your db in a uniquely identifying manner.


In reaction with the other answer, let try fuzzywuzzy with the example you give.

>>> from fuzzywuzzy import fuzz
>>> a = "Shanghai, China"
>>> b = "China, ShangHai"
>>> fuzz.ratio(a, b)
53
>>> fuzz.partial_ratio(a, b)
53
>>> fuzz.token_sort_ratio(a, b)
100
>>> fuzz.token_set_ratio(a, b)
100

seems fair.

$\endgroup$
  • $\begingroup$ Any question @eddie ? $\endgroup$ – keepAlive Oct 17 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.