# Finding the top words in a text

(I have no actual background in data science or statistics, so please easy with the math and concept names).

I have a text file (lets say 40k words), and I want to find the top 10 words with the highest weight (excluding the too common "the, at, a, ...").

I figured I'll split the data to (let's say) 200 chunks, and calculate the tf-idf of each word.

I did this in Python. I got a 200 x 10k matirx (200 chunks, 10k words). Each word has it's TF-IDF calculated in it's row (chunks).

This allows me to calculate the top 10 words only of one chunk in comparison to the others, but I can't really calculate the total tf-idf of each word. If I just sum the columns, it doesn't help.

This is the post in Stack Overflow: https://stackoverflow.com/q/46694163/7252805 (nobody answered).

I'd love some help.

• What do you mean by "total TF-IDF"? It is a relative measure, that is context dependent, so "total" here is unclear.
– Tim
Oct 20, 2017 at 13:41
• maybe my train of thought was wrong. I'm trying to find the top 10 words within a text file, without words like "the, a, at, ...". Oct 20, 2017 at 13:44
• Then you can simply sort the words by frequency and set up some cut-off that removes the stop-words. Alternatively, use one of the multiple available stopwords lists (python libraries for text processing have them implemented as far as I remember) and just remove them.
– Tim
Oct 20, 2017 at 13:53