3
$\begingroup$

I'd like to compute a TFIDF matrix (tfidf_matrix_b) based on a previously computed TFIDF matrix (tfidf_matrix_a). Is there a good way to do this using prebuilt R functions / a good algorithm to do so? I am aware of packages such as tm (in fact, it is my difficulty in using tm that prompted this post). I can certainly write a function for myself that will:

1) Calculate tf_matrix_a from corpus_a

2) Calculate tf_matrix_b from corpus_b using a dictionary created from corpus_a

3) Calculate tfidf_matrix_a and tfidf_matrix_b by dividing tf_matrix_a and tf_matrix_b by the document frequencies calculated from tf_matrix_a

However, I would have thought there would be a more elegant solution / this already existed in something like tm. Does anyone have a more elegant solution?

If it's helpful, this is for the purposes of text classification, classifying new data based upon a model trained on old data. Gotta have the new data in same form as old data to do so!

Thank you!

$\endgroup$
1
  • $\begingroup$ To stave off closure votes based on the impression that you are just asking for code, please consider editing this post to make it clear that your question is more general than that and your "more elegant solution" may be an algorithm or formulas, not just R code. $\endgroup$
    – whuber
    Commented May 27, 2015 at 22:20

1 Answer 1

1
$\begingroup$

In broad terms, whether in R, python, or a third language, you want to use the same fitted TFIDF transformer (both the vocabulary and the IDF component) on new data. In python, this is very easy (see scikit-learn). In R, with tm, say, it's easy to specify vocabulary, but you'll probably need to save this vocabulary and the IDF yourself somehow. Hope this is helpful for anyone else looking at this problem and that the broad point (and not specifics of programming) are also clear.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.