A measure for sustained activity (dispersion)

Question

I need a numeric measure that determines a level of sustained activity.

Let me describe a situation in which this could be useful. Imagine that you have a Twitter feed in which you are tracking words that do not appear in the dictionary. The purpose being that you're trying to figure out if there any new hip words that people are using. So not only does a word need to be frequently used it needs to have sustained usage.

A simple way of doing this is to record summary statistics for each word. So the count of the word indicates popularity and the standard deviation of the time stamps of a word usage could indicate sustained usage of that word. Or perhaps kurtosis could?

One issue however with standard deviation is that if a word is very very popular it tends to narrow the standard deviation, so that might not be the best indicator of usage.

If a word is used very infrequently the standard deviation is bloated and would overemphasize usage.

Any recommendations for a metric or measure?

Answer 1

There are actually two questions that you are trying to answer. The first is measuring frequency, which can be done comparing standard deviations or by looking at the counts in terms of a Poisson distribution, and the second question is around the idea of sustainability. This means you need two measures. The standard deviation is a good measure for frequency.

"How sustainable something is" is a difficult thing to comprehend. It has a lot to do with entropy. Essentially, you need to be able to measure how a word disperses through Twitter. The best way to do this is to capture how many people are using the word, use a geographic profile to see how widespread the word becomes and to see how quickly it spreads, and also use contextual analysis to see what the word means and how it is being used. If it can be used broadly, then the word will probably be used more frequently and probably last longer than a word that is topic-specific. Those are some basic components, but sustainability is a concept based around entropy in that things tend to disorder in an isolated system as time goes on. Using Twitter pretty much eliminates the "isolated system" condition.

Many words tend to develop from hot button issues and new paradigms. That is why context is so important in understanding how sustainable a word is.

Finally, you may also want to see how a word becomes a word in the dictionary. That should tell you something of its popularity and usage. Here is the overview from Merriam-Webster: http://www.merriam-webster.com/help/faq-words-into-dictionary.