# Chi-squared test for detecting trending terms

I'm trying to find bursty(trending) terms in a text stream. There are two frame in a stream; expected frame and observed frame.

For each frame, I tokenize documents(tweets/blog posts etc.) up into terms(words), so I have two term-frequency lists:

For Expected Frame:

• term a, 2500
• term b, 2000
• term c, 1500
• ...
• term m, 23
• term x, 11

For Observed Frame:

• term a, 2600
• term b, 1900
• term m, 1500
• ...
• term x, 15

Now, I need a method to find 'term m' as a bursty term. I think chi-squared test can tell which terms is bursty, in other words which term has changed significantly.

As you can see, I'm not interested in detecting whether whole observed frame is changed or not, but each individual term.

My questions is that does chi-squared test fit this problem and what other methods are there?

If I use chi-squared, what is the critical value for 'term m', how can I calculate? "chi-value= power( 1500-23 ,2) / 23" is ok?

• What do you mean by expected frame? I can understand that by "observed frame" you are tracking the occurrences of the token, but how did you calculate the expected frame? I hope if you can answer because I need this for my application as well? Jul 9, 2013 at 8:21

$\sum{\frac{(observed-expected)^2}{expected}}$
$\frac{(observed-expected)}{\sqrt{expected}}$