How to characterize a problem of standardizing product descriptions I'm looking for some advice for where to start on this problem. Let's say I have sales transaction data from a number of different retailers that all sell the same products. Even though they are selling the same products they all identify them a little bit differently. For example for one particular product one retailer calls it "Kellogs Corn Flakes 20oz." and another calls it "Kel CF, 20". And there are other representations for the 100's of other retailers.
The problem to solve is how to map each of the different products to a standard set of product descriptions so that the data can be aggregated. Assume that I have some group of retailers already mapped manually.
I've been going over in my head where to start with this. Is it a search problem where I consider the search query is the retailer's representation and the standard description is the "document" to find? Or is it a classification problem in that I'm trying to classify each description into a standard product description category? Or can named entity recognition play into it somehow.
Any advice you could provide to get me started would be much appreciated. I've been looking at tools like Apache Lucene, Solr and OpenNLP but it's just not clear to me how to characterize the problem.
 A: I'm not sure this question can be answered in any kind of affirmative way. But what you have in front of you is a record linkage and/or entity resolution task.
This is how I would approach the problem:


*

*Define some kind of similarity measurement between descriptions (e.g. cosine distance on tf-idf vectors)

*Use some simple criteria to eliminate "no way they are a match" possibilities (e.g. don't even bother to compare items measured in kilograms with items measured in inches) -- this is called "blocking" and it makes the next step much less intensive computationally

*Within the blocks defined above, compute the similarity between all pairs of records

*Discretize that similarity measurement into "clusters" of individual entities. You can either use a cutoff (all records with score above some threshold), a supervised statistical or machine-learning model (predict match/no-match by fitting a model on labeled pairs of records), or some clustering algorithm (e.g. HDBSCAN or agglomerative hierarchical clustering)


The docs for the Python library Dedupe go into some detail on this procedure and might give additional hints at a starting point. You will probably also encounter allusions to the Fellegi-Sunter model for record linkage, but it's kind of a meta-model, I don't personally think it's all that helpful, and I'm not sure it's appropriate in your case.
