I'm analyzing newspaper articles, some are 2 or 3 pages long, some a few lines long.
Corpus is made of a few thousands of dated articles spanning over a few years.
My goal is to detect
trending terms at a monthly level, and to compare
term occurrence from month to month.
It's easy to assign a
tf-idf to an article but I struggle as to how to assign one to a month.
If I just consider a month as a big document, my
idf will most of the time be null, at least considering the usual definition
idf = log(N/nt) as pretty much all the words that would be relevant are used every month.
idf of a term computed at the article level is a better indicator of specificity so I thought of computing the
tf-idf by multiplying the monthly
tf (sum of article level
tfs) by the article level
Another approach would be to take the average article level
tf-idf for a given month and term.
Both make sense conceptually but I'm thinking it must be a common problem and some better suited methods have probably been developed and researched (and I won't have the ressources to calibrate my method much so I'd better tick to a tried and proved generic method).
I understand the issue may not be properly defined for one unique objective answer but I have to start somewhere.
Can you give me some pointers ?