2
$\begingroup$

The sklearn implementation of TFIDF allows you to specify minimum & maximum token frequencies as a precursor to calculating the IDF weighted scores. It will first create a count matrix and prune out the tokens not meeting your criteria. The default setting allows for all tokens to be used (eg. min 1, max 100%).

I understand why you would want to adjust these variables when calculating TF alone. However, why adjust this when using TFIDF? Doesn't the IDF weighting already account for what these variables attempt to accomplish?

I often see examples of TFIDF where people tweak these parameters. For instance: min 2 documents, max 90%. Other's might do 5%/95%. Sometimes people don't make any adjustments at all.

Should these parameters be treated like other parameters in ML algorithms (adjust/test), or is there a logic behind adjusting them in TFIDF?

Edit - Additional Information

What I did was run a simple loop through NLTK's Reuter's dataset for "oil" news articles. Each loop took the max_df function from 1.0 to 0.1 by 0.5 increments. I called the transform each time and took the average TFIDF score across all documents for that specific term. Shown below are the top and bottom 10 words for the corpus ranked by their IDF score.

It seems to me now that the most drastic impact would be to remove a handful of words likely already taken care of by the stop_words argument. The IDF score won't change, and there seems to be a minimal change due to how sklearn normalizes the output.

Top Bot

$\endgroup$
  • $\begingroup$ Sorry to bump this, but did you ever find out or does anyone know? $\endgroup$ – francium87d Jun 20 '17 at 9:38
  • 1
    $\begingroup$ I added some additional info. What I did was just run the sklearn tfidf function in a loop for values of max_df from 0.1 to 1.0. The values change slightly due to the normalization process I think. Otherwise it seems this parameter would have minimal impact in the grand scheme of things. As most of what it does is remove popular words, likely taken care of by the stop_words argument. Though "oil" is not a stopword and would be a good idea here for max_df to be under 95% to remove it. $\endgroup$ – ElPresidente Jun 21 '17 at 4:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.