# Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate idf , rather than td-idf as in sklearn?

• I am not very familiar with this package but isn't the transform() call doing just this? The fit() is done on training data but transform() on test... – Vladislavs Dovgalecs Jul 26 '15 at 5:59
• I think no. It first transform the corpus into frequency data, then fit_transform() to td-idf matrix. But in test set it may have words that are not in train set, so fit_transform() won't work. I just tried. – user83176 Jul 26 '15 at 7:13
• I see. it may work – user83176 Jul 26 '15 at 7:16
• You could just save the df from the train set and use it on the test set. I use R, so can't speak to Python, but that's one approach that's not that hard. – shf8888 Jul 28 '15 at 14:34

As for learning the idf, sklearn's TfidfVectorizer learns the idf after fit or fit_transform method call and is available as an attribute of the classifier:
clf = sklearn.feature_extraction.text.TfidfVectorizer()