say I have N WIKI pages and I want to find the words within these pages that are characteristic of each page as compared to the others. As suggested here previously, the standard approach would be to use tf-idf to find these words. say I did just that, now I want to statistically test that these words are indeed enriched in one page over the others, what is the statistic for that ? the rank of of the word on the tf-idf ? should I do sort of a post-hoc test using the proportion of these words in the various pages?

and an additional related question; say I want to group the words and test that they are enriched in one page over compared to the others (e.g. "hello", "Hi", "dear" etc. are grouped) could I use hyper-geometric distribution to calculate the probability of k occurrences of one or more of these words in an N words page and then compare the computed probabilities between the different pages ?

help is appreciated!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.