1
$\begingroup$

Let say I have a list of words, such as:

apple
apale
aaple
apples
oranges
ornnges
orange
orage
melons
meeons
meeon
melon
melan

I want to group them based on similarity (or maybe I should say cluster them). Obviously, from above list, there are three groups: apple, orange, melon. Do you have any idea on how to achieve this (in machine learning or statistical sense)?

Thank in advance.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ All you need is a simple clustering algorithm (e.g., k-means, k-means++, k-medoids/PAM, DBScan, ...) where instead of a typical distance function (euclid, manhatten, etc.) you should use a string edit distance. Did you already looked up these? en.m.wikipedia.org/wiki/Edit_distance $\endgroup$
    – NeuroMorphing
    Commented Jul 9, 2016 at 22:51
  • $\begingroup$ Got it. I found a better way to do this. $\endgroup$
    – RockTheStar
    Commented Jul 12, 2016 at 19:17
  • $\begingroup$ Let me add more information to it. As suggested, we use clustering to achieve this, which is by calculating the distance of each word. So, what I did is to use word similarity calculation by en.wikipedia.org/wiki/Levenshtein_distance, as what @roundsquare suggest :). There are also other good word similarity matrices outthere. $\endgroup$
    – RockTheStar
    Commented Mar 12, 2018 at 1:11

3 Answers 3

Reset to default
0
$\begingroup$

Some of what you want to do can be done via lemmatization. I've used the WordNet Lemmatizer in NLTK and is helpful. You can look at: http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization which describes it well.

Improve this answer
$\endgroup$
2
$\begingroup$

Don't do this. It produces too many false matches.

dig
dog
fog
fag
fap
nap

all differ by one letter from one to the next. They would "cluster" if you do not use corpus information.

Improve this answer
$\endgroup$
1
$\begingroup$

What language do you use?

If you understand Python take a look at: https://wakari.io/sharing/bundle/iuliacioroianu/Text_analysis_Python_NLTK

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.