2
$\begingroup$

I would like to have some advice on the way to calculate burstiness. I am working with a set of text data, where every term is calculated with their frequency in newspaper for 2 weeks, e.g. "apple" during their iphone4s release will be day1 = 10, day2 = 300, day3 = 25, and so on till day14. So if popular term such as Obama which appear day1 = 100, day2=100, day3=105 ... since it is popular but some other terms which is bursty and spiky will be like "apple" example.

Is there any way that we can measure such burstiness, I am aware of standard deviation, and is there any other ways? The ultimate goal is to make burstiness(Obama) as small as possible and the burstiness(apple) as high as possible. Thanks.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Try Index of Dispersion, i.e. variance/mean ($D = \sigma^2 / \mu $). It is also known as Fano factor (for your case - with the time window $W=1\text{ day}$).

For Poisson distribution it is just $D=1$ (e.g. if people write post all the time, not related to each other).

For bursty events it is $D>1$ (e.g. there is hype about a product by Apple).

For 'self-avoiding' events $D<1$ (e.g. daily reports; if each day there is the same number of posts, it is just $D=0$).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Interesting idea. It seems that "negative bursts" where the frequency suddenly goes down would also be detected this way. It's not clear whether this is intended, but the multiple references to "popular" in the question suggest not: a burst is solely a sudden temporary increase in the frequencies. $\endgroup$
    – whuber ♦
    Commented Jan 8, 2012 at 17:02
1
$\begingroup$

A single-day burst will appear as an outlier in the variable's distribution. There are some established ways to assess whether values are outliers--besides calculating by how many standard deviations they differ from the mean. One is Dixon's Test (see the notes here).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is Dixon test just used to detect a single outlier? Or you are suggesting to calculate number of outliers as burstiness (i.e. more outliers means more bursty?) $\endgroup$
    – drhanlau
    Commented Jan 8, 2012 at 14:32
  • $\begingroup$ The original ratio test is designed to detect a single outlier. If there are two outliers it suffers from the masking effect that many outlier detection tests suffer from. To get around this Dixon modified his test statistic. Instead of [X(n-1)-X(1)]/[X(n)-X(1)] for two outliers he uses {X(n-2) -X(1)]/[X(n)-X(1)]. This eliminated the masking effect that the second outlier has in hiding the first. $\endgroup$
    – Michael R. Chernick
    Commented May 10, 2012 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.