0
$\begingroup$

I have a data set with some phrases.

samp$reviewText[1:5]
[1] "good use worked expected"                                
[2] "brother got screen protector nice finish"                
[3] "however bubbles really hard get also got huge rip middle"
[4] "never stay stuck phone"                                  
[5] "use iphone good product drop phones save glass"   

My objective is to build clusters of phrases with a similar meaning.

I execute this task by running the following operations.

Converting the phrases to a bag of words representation.

library(tm)
corpus = Corpus(VectorSource(samp$reviewText))
dtm = DocumentTermMatrix(corpus, control = list(weightTfIdf))
dtm =  as.data.frame(as.matrix(dtm))
colnames(dtm) = make.names(colnames(dtm), unique = T)

Perform the clustering (I use 3 centers in this example) and merging the clusters to samp.

clust = kmeans(dtm, 3)
samp$cluster = clust$cluster

So the result would like this:

samp_cluster[1:5,]
                                                reviewText cluster
1                                 good use worked expected       3
2                 brother got screen protector nice finish       1
3 however bubbles really hard get also got huge rip middle       3
4                                   never stay stuck phone       3
5           use iphone good product drop phones save glass       3

Is this the right way to do this task?

$\endgroup$
1
  • 1
    $\begingroup$ Why would you use k-means for this? K-means is a poor clustering algorithm in general, & shouldn't be used for non-normal data in particular (binary data, has this word / doesn't, are not normal). $\endgroup$ Commented Jan 30, 2019 at 14:20

1 Answer 1

3
$\begingroup$

Using bag of words for finding similar sentences doesn't (at least shouldn't) work well because you don't capture any semantics. Two phrases having the same meaning but with totally different words will not be similar. If they fall into the same cluster, I believe it'll be by chance. In order to capture meanings, you should use something like word-embeddings, where each word is mapped to a $K$ dimensional space and similar meaning words tend to be closer. Thus, when you construct numerical vectors for phrases (one simple way, by averaging the embeddings of each word in the sentence), and then apply a clustering algorithm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.