I have a list of products, including variables such as the product name (as it appears on the receipt) and the merchant where the product was bought.
I have a good deal of them manually classified into a fixed group of categories (e.g. alcoholic drinks, vegetables, meat, etc.).
The data are, as always, noisy. In this case, particularly noisy because it comes from scanned receipts and OCR on not so good scans is usually very noisy.
I want to play around with algorithms to classify new data, using the two variables above.
There are several major sources of variation here:
- The OCR, which means a product (e.g. chicken) can be found with many different but relatively similar spelling (e.g. chiken, hicken, chicen, ...).
- The same product can have different names, according to the merchant who sold the product. In this case, the names can be either similar or completely different across merchants, but rather similar within every merchant.
- The same product can have very different names within the same merchant (e.g. branded products whose name on the receipt is the brand name, vs. generic names; soft drink vs. coca cola).
I've tried some (kind of naive) classifier, using for example the distance between strings (which tackles mostly the first main source of variation mentioned above), but I am not very happy with the results.
So I wanted to reach out here to ask for ideas on how to tackle this problem. I guess many people have "solved" or at least worked much longer in this kind of problem than I did (a few hours) so I would really appreciate any guidance here.
By the way, I use mostly R, so R-based solutions would be greatly appreciated.