0
$\begingroup$

I have this model where I have a count of a word. Every day I do a count of the word and then calculate a simple ratio for this word by saying:

Ratio = Current day count / Past day count

So I now have a lot of counted words and the variables Ratio and Count.

I now wan't to find the words that are trending, those that are trending but going down, The average words, the suddenly words, the never mind words.

I will explain those:

  • The trending words are those words with a high Ratio and a high count.
  • Those that trending, but going down is the words that have a high count but getting lower on the ratio.
  • The average words are like the words with a mean value of the ratio and counts.
  • The suddenly words are the words with a high ratio but low count.
  • The never mind words are those with low ratio and low count.

I am not a master in statistics (never done any classes) but I was thinking of using a K-Means algorithm on the data set to split it into 5 clusters, where each cluster represents one of the categories. Is this a way to use K-Means or am I completely off track?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

A few things about K-means suggest that you may be thinking about it incorrectly:

  1. K-means cluster analysis is an unsupervised process so you will not be able to determine the groups you end up with.
  2. The algorithm works by you choosing some variables to use for discriminating groups; the distance between each observation on these is what will make the clusters.

Based on your example you are using the count and ratio but some predicted clusters also incorporate the trend over time how are you thinking of including this?

That said, I think you may be able to use k-means to validate the clusters you hypothesize. Depending on the software and function you are using you may be able to choose starting values that are close to your predicted groups. This will be helpful for because k-means will otherwise choose random numbers. The output from the algorithm should allow you to reproduce the mean values on the features you select or prototype (example) points from each cluster. If these points follow your predicted clusters it is evidence that your pattern is correct.

One final note is that because you'll be selecting the variables used in separating clusters, you shouldn't use the same variables in follow-up analyses to validate the groups. What I mean, for example, is that you shouldn't say that one group has a significantly higher ratio that another because it probably will by nature if the algorithm. If you divide low ratios and high ratios into two groups, they will be different.

Improve this answer
$\endgroup$
3
  • $\begingroup$ As you said, I won't be able to determine the groups I will end up with. Is this more like a supervised problem than a unsupervised problem? $\endgroup$
    – McBoman
    Commented May 25, 2016 at 6:22
  • $\begingroup$ I'm not familiar enough with supervised machine learning to confidently say so. That is my feeling though. Like I mentioned in my answer, you could still see if the pattern of clusters is actually present by comparing the k-means results to your hypothesis though. $\endgroup$
    – Andrew Jackson
    Commented May 25, 2016 at 10:39
  • $\begingroup$ I think I just understood what you meant. What if I put the ratio and count in a k-mean calculation, then save the clusters as a value on the schema so for the next computation I have the following values: Ratio, Count, Current Cluster. Then I can time series the data and look at if the cluster of the word change over time. Does that make sense? $\endgroup$
    – McBoman
    Commented May 25, 2016 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.