0
$\begingroup$

I am working on optimizing ~8000 financial and operational reports which have frequency ranging from monthly, quarterly and yearly.

To accomplish this I am using text mining to identify similar and near-duplicate reports (i.e. reports giving out same information have been created more than once by different analysts).

I have basic knowledge of text mining and trying to do it in R.

Can anyone help me with appropriate R packages and methodologies?

$\endgroup$

1 Answer 1

0
$\begingroup$

A basic infrastructure for text mining is provided by the tm package. It allows you to perform all the usual step of a text mining analysis: text parsing and cleanse, stemming, and finally building the term-document matrix (with different weights).

From my point of view the term-document matrix is the starting point of your analysis. Starting from that one you can apply any sort of distance computation to find correlated articles. Articles with very low distance are candidates for being duplicates.

This is one of the main pages that will drive you to an analysis: https://cran.r-project.org/web/views/NaturalLanguageProcessing.html

Hope this helps

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.