I am working on a project and need to extract persons' names from a large amount of documents. This task should belong to the named entity resolution problem. What are currently some of the most popular open source software/libraries to perform the named entity resolution?
The problem of named entity resolution is referred to as multiple terms, including deduplication and record linkage. I doubt that it is possible to determine precisely, what software belong to some of the most popular for solving that problem. There are various approaches and algorithms can be used for named entity resolution. Therefore, software which implements those can be seen as complementary to each other (perhaps, there exist multiple research studies that compare and benchmark entity resolution approaches and algorithms, but so far I have seen only two of them - see references below, denoted with a triple asterisk "***").
This nice tutorial (in a form of presentation slides) on entity resolution provides a comprehensive overview of the problem and the solutions, including both approaches and algorithms. The tutorial also provides an extensive set of references to sources with further information. Speaking about corresponding software, one may find open source or dual-license projects, such as Java-based Stanford NLP Group software (which includes Stanford named entity recognizer (NER)), Stanford Entity Resolution Framework (SERF), LingPipe (which includes a NER module) and Duke library, as well as Python-based NLTK software (http://www.nltk.org/book/ch07.html). I realize that named entity recognition and resolution are quite different tasks, however, some of the above-referenced software, focused on the former, might be useful for the latter, by using appropriate code segments.
Additionally, the following IMHO related/relevant software and papers might also be of interest:
- Information Extraction framework in Python;
- GATE software, in particular, ANNIE information extraction system;
- several other NER tools, mentioned in this paper***;
- this excellent overview*** of NER approaches, including neural networks and deep learning;
- Ontotext's S4 (Self-Service Semantic Suite) on-demand software provides access to linked data repositories, such as DBpedia, Freebase and GeoNames;
- Elasticsearch NER plug-in for Duke;
- this paper on Swoosh algorithms, implemented by SERF software;
- Wikilinks Corpus, released by Google;
- this paper on entity disambiguation;
- book "Data Matching" on record linkage, entity resolution, and duplicate detection.
Be careful not to confuse "entity resolution" with "named entity recognition". Named entity recognition refers to finding named entities (for example proper nouns) in text. That's what your original question asked for. You can do this in NLTK & Python for example, or using Stanford's NER tool.
Entity matching (or entity resolution) is also called data deduplication or record linkage. This is not the same thing as NER. Entity matching says, "given these two entities, are they the same thing?" To match entities, we may look at features about them, for example if we are trying to decide if Mary Smith is the same as Mary Q. Smith, we might check to see if the street addresses are the same for both people. We can also use string distances or other metrics (Levenshtein, soundex, etc) on the names themselves to see if they are "close enough" to possibly represent the same person.
But entity matching and named entity recognition are NOT the same thing.
I've come across DataMatch by Data Ladder, which is a great fuzzy matching and entity resolution tool used across business and would work really well for this situation. They offer a complimentary trial for new users.
In fact, an independent verified evaluation was done of the software comparing it to major software tools by IBM and SAS. There was a study done at Curtin University Centre for Data Linkage in Australia that simulated the matching of 4.4 million records. It identified what providers had in terms of accuracy (Number of matches found vs available. Number of false matches)
1. DataMatch Enterprise, Highest Accuracy (>95%), Very Fast, Low Cost 2. IBM Quality Stage, High Accuracy (>90%), Very Fast, High Cost (>$100K) 3. SAS Data Flux, Medium Accuracy (>85%), Fast, High Cost (>100K)
Ralph, These are deduplicaton tools for data ETL, IBM's Quality Stage is a data deduplication tool mainly used during processing data during typical data ETL, it is not an Entity Resolution tool. There is a difference between the two concepts. For entity resolution, try IBM InfoSphere MDM or IBM InfoSphere Identity Insight.