3
$\begingroup$

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate idf , rather than td-idf as in sklearn?

$\endgroup$
4
  • 1
    $\begingroup$ I am not very familiar with this package but isn't the transform() call doing just this? The fit() is done on training data but transform() on test... $\endgroup$ Jul 26, 2015 at 5:59
  • $\begingroup$ I think no. It first transform the corpus into frequency data, then fit_transform() to td-idf matrix. But in test set it may have words that are not in train set, so fit_transform() won't work. I just tried. $\endgroup$
    – user83176
    Jul 26, 2015 at 7:13
  • $\begingroup$ I see. it may work $\endgroup$
    – user83176
    Jul 26, 2015 at 7:16
  • $\begingroup$ You could just save the df from the train set and use it on the test set. I use R, so can't speak to Python, but that's one approach that's not that hard. $\endgroup$
    – shf8888
    Jul 28, 2015 at 14:34

1 Answer 1

3
$\begingroup$

You should always use the deflators from your training data (same goes for if you standard scale your data for Support Vectors or Neural Networks).

As for learning the idf, sklearn's TfidfVectorizer learns the idf after fit or fit_transform method call and is available as an attribute of the classifier:

clf = sklearn.feature_extraction.text.TfidfVectorizer()
clf.fit(mydata)
clf.idf_  # this is the idf data
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.