I'm trying to find bursty(trending) terms in a text stream. There are two frame in a stream; expected frame and observed frame.
For each frame, I tokenize documents(tweets/blog posts etc.) up into terms(words), so I have two term-frequency lists:
For Expected Frame:
- term a, 2500
- term b, 2000
- term c, 1500
- ...
- term m, 23
- term x, 11
For Observed Frame:
- term a, 2600
- term b, 1900
- term m, 1500
- ...
- term x, 15
Now, I need a method to find 'term m' as a bursty term. I think chi-squared test can tell which terms is bursty, in other words which term has changed significantly.
As you can see, I'm not interested in detecting whether whole observed frame is changed or not, but each individual term.
My questions is that does chi-squared test fit this problem and what other methods are there?
If I use chi-squared, what is the critical value for 'term m', how can I calculate? "chi-value= power( 1500-23 ,2) / 23"
is ok?