1
$\begingroup$

Let say I have a list of words, such as:

apple
apale
aaple
apples
oranges
ornnges
orange
orage
melons
meeons
meeon
melon
melan

I want to group them based on similarity (or maybe I should say cluster them). Obviously, from above list, there are three groups: apple, orange, melon. Do you have any idea on how to achieve this (in machine learning or statistical sense)?

Thank in advance.

$\endgroup$
3
  • 1
    $\begingroup$ All you need is a simple clustering algorithm (e.g., k-means, k-means++, k-medoids/PAM, DBScan, ...) where instead of a typical distance function (euclid, manhatten, etc.) you should use a string edit distance. Did you already looked up these? en.m.wikipedia.org/wiki/Edit_distance $\endgroup$ – Unhandled exception Jul 9 '16 at 22:51
  • $\begingroup$ Got it. I found a better way to do this. $\endgroup$ – RockTheStar Jul 12 '16 at 19:17
  • $\begingroup$ Let me add more information to it. As suggested, we use clustering to achieve this, which is by calculating the distance of each word. So, what I did is to use word similarity calculation by en.wikipedia.org/wiki/Levenshtein_distance, as what @roundsquare suggest :). There are also other good word similarity matrices outthere. $\endgroup$ – RockTheStar Mar 12 '18 at 1:11
0
$\begingroup$

Some of what you want to do can be done via lemmatization. I've used the WordNet Lemmatizer in NLTK and is helpful. You can look at: http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization which describes it well.

$\endgroup$
1
$\begingroup$

What language do you use?

If you understand Python take a look at: https://wakari.io/sharing/bundle/iuliacioroianu/Text_analysis_Python_NLTK

$\endgroup$
0
1
$\begingroup$

Don't do this. It produces too many false matches.

dig
dog
fog
fag
fap
nap

all differ by one letter from one to the next. They would "cluster" if you do not use corpus information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.