New to R, and am trying to do text classification. I am using R package tm to convert raw txt data into matrix. Here's the relevant code snippet.

 col <- Corpus(DirSource(path),
                readerControl = list(reader = readPlain,
                language = "en",
                load = TRUE)))

 tdm <-  TermDocumentMatrix(col, control = list(tokenize = NGramTokenizer))

I have the following questions:

1) Feature selection

I need to do chi-squared or information gain based feature selection on my data. Which R packages can I look at? I came across at caret and boruta but they do not seem to be appropriate for what I am wanting to do.

2) Handling new (unseen) instances

Let's say I have trained my model using my training set. When the test set comes in, I would need to pass it through same filters (stemming, stopword removal, tf-idf weighting, feature selection etc.). I have no idea how to do this !

Any hint/help will be much appreciated.

Thanks in advance.


closed as not a real question by user88 Jul 31 '11 at 14:12

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ There is an answer already, so I'll close it for now so user4581 had a chance to access his answer when possibly answering your new questions. Then it will be deleted as you asked. $\endgroup$ – user88 Jul 31 '11 at 14:15

Regarding feature selection, as you described, depends on the text content ! How do you classify them manually ?! Because of some impression or words ?!(negative - positive) (happy - sad - neutral). Frequency of words could be also informative. check out this package.

There are several review and papers like this on net.

You can define your own kernel check kernel methods, in order to put similar text together and distinguish dissimilar. check string kernels from kernlab and here

For implementation for text classification in R, look at this list and find your interest method and package.

To address your second question, I can say, do everything(Feature extraction - and all other you mention) + label of the class. Then choose part of data, say 70% and make your own model. Afterward, run your model on the unseen data(30%) and check its output with actual label. then you simply can calculate the accuracy of model.

  • $\begingroup$ Thank you for your answer. As mentioned in my question, I am in fact using the tm package for preprocessing my data. Actually what I was wanting to know is (1) If there are off the shelf tools for feature subset selection (2) How the transformations (like tf-idf for example) would be applied to TEST set (for example: tf-idf computations on test set would still need to be done on the scale of the training set) - I was hoping to see some code snippets to help with this. But anyway, now I think I will pose these questions separately. $\endgroup$ – Andy Jul 31 '11 at 19:55
  • $\begingroup$ Some keywords like "dimension reduction", "subset selection", could be helpful to look through here, before throwing question ... and still I am not sure if I get your second question properly ... $\endgroup$ – user4581 Jul 31 '11 at 20:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.