New to R, and am trying to do text classification. I am using R package tm to convert raw txt data into matrix. Here's the relevant code snippet.
col <- Corpus(DirSource(path), readerControl = list(reader = readPlain, language = "en", load = TRUE))) tdm <- TermDocumentMatrix(col, control = list(tokenize = NGramTokenizer))
I have the following questions:
1) Feature selection
I need to do chi-squared or information gain based feature selection on my data. Which R packages can I look at? I came across at caret and boruta but they do not seem to be appropriate for what I am wanting to do.
2) Handling new (unseen) instances
Let's say I have trained my model using my training set. When the test set comes in, I would need to pass it through same filters (stemming, stopword removal, tf-idf weighting, feature selection etc.). I have no idea how to do this !
Any hint/help will be much appreciated.
Thanks in advance.