i extract and rank a list of the important hashtags (using td-idf ) from the twitter dataset(twitter.csv) that just includes list of tweets and now i have 9 important hashtags, now i want to use those important hashtags as nodes and cluster(preferably dendrogram) all the tweets.

explanation: i first came up with a character class that include all hashtags in the twitter data,then i use Corpus for text mining it, then i save them into a Term Document Matrix (a.tdm), and further i came up with the list of important hashtags.

my problem is i can not find a way to use those hashtags for clustering the whole data, i am trying the code below to cluster but it only clusters those hashtag names.

a.tdm.sp <- removeSparseTerms(a.tdm, sparse=0.989)
mydata.df <- as.data.frame(inspect(a.tdm.sp))
mydata.df.scale <- scale(mydata.df)
d <- dist(mydata.df.scale, method = "euclidean")
fit <- hclust(d, method="ward")
groups <- cutree(fit, k=5)
rect.hclust(fit, k=5, border="red")

i hope i could have explained enough.

  • $\begingroup$ You could create an nxp similarity matrix where the p's are the important hashtags and the n's are all of the other hashtags, which would treat the important hashtags as seed input to a hierarchical clustering routine. $\endgroup$ Nov 25, 2015 at 18:51
  • $\begingroup$ thank you, but how can i create a similarity matrix that use important hashtags a p and other hashtags as n, because i don't understand why there should be any similarity between important hashtags and other hashtags ? $\endgroup$
    – ramin
    Nov 25, 2015 at 19:00
  • $\begingroup$ Are you saying there's no association between the "important" hashtags and the others? If that's so, then what were you hoping to base an analysis on? $\endgroup$ Nov 25, 2015 at 19:03
  • $\begingroup$ yes i am not looking for relationship between other hashtags and important hashtags , i want to use those important hashtags and cluster the tweets that are having those hashtags inside them $\endgroup$
    – ramin
    Nov 25, 2015 at 19:05
  • $\begingroup$ Ok...then replace the n hashtags with tweets. $\endgroup$ Nov 25, 2015 at 19:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.