We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate idf , rather than td-idf as in sklearn?

  • 1
    $\begingroup$ I am not very familiar with this package but isn't the transform() call doing just this? The fit() is done on training data but transform() on test... $\endgroup$ – Vladislavs Dovgalecs Jul 26 '15 at 5:59
  • $\begingroup$ I think no. It first transform the corpus into frequency data, then fit_transform() to td-idf matrix. But in test set it may have words that are not in train set, so fit_transform() won't work. I just tried. $\endgroup$ – user83176 Jul 26 '15 at 7:13
  • $\begingroup$ I see. it may work $\endgroup$ – user83176 Jul 26 '15 at 7:16
  • $\begingroup$ You could just save the df from the train set and use it on the test set. I use R, so can't speak to Python, but that's one approach that's not that hard. $\endgroup$ – shf8888 Jul 28 '15 at 14:34

You should always use the deflators from your training data (same goes for if you standard scale your data for Support Vectors or Neural Networks).

As for learning the idf, sklearn's TfidfVectorizer learns the idf after fit or fit_transform method call and is available as an attribute of the classifier:

clf = sklearn.feature_extraction.text.TfidfVectorizer()
clf.idf_  # this is the idf data

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.