Input to tfidf is a list of documents, output is a matrix which contains a numerical value for each (word, document) pair. How can I use that matrix to perform feature selection, i.e. reduce the size of the dictionary?

  • $\begingroup$ One possible option is to use a model with l1 regularization $\endgroup$ Aug 22 '16 at 5:01

Normally, feature selection is done against some criteria, that is, select all features that best separate/describe your class/target function. That being said, you can select features using their variance or co-variance. In your case, that would mean to only select words (or documents) where the (normalized) TF-IDF score varies the most among documents (or words). In the co-variance case, you'd remove redundant features, that is, remove words that have similar TF-IDF among all documents they occur in (by calculating a correlation coefficient between each word "row" in your TF-IDF matrix), typically keeping the word (document) with the highest frequency, i.e., row (column) sum.

However, rather than feature selection, you might also be interested in feature extraction if all you want is dimensionality reduction, in particular word embeddings (sometimes confusingly called "word representations").

Note that I provide more approaches/answers in a related question regarding feature selection of words for explicitly unsupervised machine learning problems.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.