1
$\begingroup$

Assume I have a collection of documents and I want to use tf-idf as my document weighting measure. If I want my vocabulary to be of size 100, how do I choose those 100 words in the vocabulary? Considering training data as a whole and selecting the 100 top frequent words is not going to work here since my weighting system is tf-idf, right?

$\endgroup$
2
$\begingroup$

A main use of tf-idf is to determine which terms might help you differentiate between your documents. When a term is in every document its idf = log(1) = 0. So the tf-idf for this term is always 0. Likewise terms that are only in a few documents should be more useful for classifying and have the highest idf scores and tf-idf is also higher.

So the short answer is to just sort by tf-idf in descending order and take the top 100. In my experience though you need between 5% and 20% of the terms. Even if this doesn't always produce good enough results it's fast, simple, and I use it as my "baseline" model. But quite often it's good enough for the domain I work in.

$\endgroup$
1
$\begingroup$

According to my experience, once you've calculated the tf-idf values for all words in the whole collection of documents. You may use some feature selection methods to select the most pertinent features (words). Some common feature selection methods are mutual information, information gain, chi-square, etc. You can easily find their definitions on Wiki, and also, if you like, you may refer to this paper: Feature Selection for Text Classification Based on Gini Coefficient of Inequality.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.