I have approx 450.000 strings in a csv file. It's part names, so the strings are normally based on one-three words. I need to somehow group them together in an automated way so I can work with these clusters instead of part by part.
Here are a couple of random examples from the CSV file see above linked:
CENTERING DEVIC, GEAR, PLUG, ADAPTER, ADAPTER, ADAPTER, APPARATUS PANEL A50, BRACKET, BRACKET, BRACKET MOUNT CS1000"P"HD, BRACKET MOUNT CS1000"P"HD, CABLE, SIGNAL CABLE, CABLE, SIGNAL CABLE, CAP, CAPSCREW, CARRIER, CHAIN, CHECK VALVE, CHUCK, CHUCK, CLEVIS, CONDENSER SET, COVER, CRAWLER SET 11-12TON, CYLINDER TUBE, DUST COLLECTOR, ELBOW, ELBOW, FILLER PIPE, FILTER ELEMENT, FLUID LEVEL, GAUGE, FOOT, FORGING, LARGE CAPACITY HUB, GPS-box D-rigg with cabin
Here is what I plan to do to attack the problem:
- Use Levenshtein to compare the strings. Take the first item of the array of strings and compare it to all others in the array and calculate a score for this first item.
- Lets say I only allow 10% difference between the first string and the second, third,.,.., and the rest strings in the array. I will remove the first item and the other items in the array that had only 10% difference or less compared with the first string.
- Continue with second string in the array and do the same calculation over and over again until I have walked through the whole array.
- Items that remains in the array, that doesn't fit (10% > diff), I will need to handle manual...
Is this a good approach? I have read a bit about hierarchical clustering, is this a better approach or can anyone guide me in another better direction that I suggested?