I'm analyzing newspaper articles, some are 2 or 3 pages long, some a few lines long.

My Corpus is made of a few thousands of dated articles spanning over a few years.

My goal is to detect trending terms at a monthly level, and to compare term occurrence from month to month.

It's easy to assign a tf-idf to an article but I struggle as to how to assign one to a month.

If I just consider a month as a big document, my idf will most of the time be null, at least considering the usual definition idf = log(N/nt) as pretty much all the words that would be relevant are used every month.

The idf of a term computed at the article level is a better indicator of specificity so I thought of computing the tf-idf by multiplying the monthly tf (sum of article level tfs) by the article level idf.

Another approach would be to take the average article level tf-idf for a given month and term.

Both make sense conceptually but I'm thinking it must be a common problem and some better suited methods have probably been developed and researched (and I won't have the ressources to calibrate my method much so I'd better tick to a tried and proved generic method).

I understand the issue may not be properly defined for one unique objective answer but I have to start somewhere.

Can you give me some pointers ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.