0
$\begingroup$

(I have no actual background in data science or statistics, so please easy with the math and concept names).

I have a text file (lets say 40k words), and I want to find the top 10 words with the highest weight (excluding the too common "the, at, a, ...").

I figured I'll split the data to (let's say) 200 chunks, and calculate the tf-idf of each word.

I did this in Python. I got a 200 x 10k matirx (200 chunks, 10k words). Each word has it's TF-IDF calculated in it's row (chunks).

This allows me to calculate the top 10 words only of one chunk in comparison to the others, but I can't really calculate the total tf-idf of each word. If I just sum the columns, it doesn't help.

This is the post in Stack Overflow: https://stackoverflow.com/q/46694163/7252805 (nobody answered).

I'd love some help.

$\endgroup$
3
  • $\begingroup$ What do you mean by "total TF-IDF"? It is a relative measure, that is context dependent, so "total" here is unclear. $\endgroup$
    – Tim
    Oct 20, 2017 at 13:41
  • $\begingroup$ maybe my train of thought was wrong. I'm trying to find the top 10 words within a text file, without words like "the, a, at, ...". $\endgroup$
    – sheldonzy
    Oct 20, 2017 at 13:44
  • 1
    $\begingroup$ Then you can simply sort the words by frequency and set up some cut-off that removes the stop-words. Alternatively, use one of the multiple available stopwords lists (python libraries for text processing have them implemented as far as I remember) and just remove them. $\endgroup$
    – Tim
    Oct 20, 2017 at 13:53

1 Answer 1

2
$\begingroup$

Depending on what you care about, there are two options, A or B:

A.) If you only care about the top 10 words in the entire document, you don't need TF-IDF. You can simply first remove stop words like "the" and "a" (The python library you are using should have a list of stopwords; there are several different options out there). Then, you can sort the remaining words by frequency. If you don't like the results because you feel the top words are boring, try a bigger list of stopwords.

B.) If you care about getting the top 10 words in each sentence compared to all the other sentences, then TF-IDF would make sense to use. That is because TF-IDF assumes there will be multiple "documents". These "documents" don't have to literally be separate files. The "documents" can be different sentences, tweets, etc., however you want to chop it up in the way that is most useful to you for your comparison. Each TF is the term frequency in each "document", while DF is the frequency of that term across all the "documents". So what you are doing when you use TF-IDF formulas is: you're saying that you care about a term if it occurs frequently in this document, but only if it also occurs infrequently in all the other documents.

It sounds like you are more interested in option A, from what I can tell.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.