Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am working on optimizing ~8000 financial and operational reports which have frequency ranging from monthly, quarterly and yearly.
To accomplish this I am using text mining to identify similar and near-duplicate reports (i.e. reports giving out same information have been created more than once by different analysts).
I have basic knowledge of text mining and trying to do it in R.
Can anyone help me with appropriate R packages and methodologies?
A basic infrastructure for text mining is provided by the tm package. It allows you to perform all the usual step of a text mining analysis: text parsing and cleanse, stemming, and finally building the term-document matrix (with different weights).
From my point of view the term-document matrix is the starting point of your analysis. Starting from that one you can apply any sort of distance computation to find correlated articles. Articles with very low distance are candidates for being duplicates.
This is one of the main pages that will drive you to an analysis: https://cran.r-project.org/web/views/NaturalLanguageProcessing.html
Hope this helps
