I am new to text processing. Currently I am trying to determine which type of feature vector I need for a classification problem. I am mainly deciding between binary feature modeling and statistics-based approaches, such as term frequency/inverse document frequency (tf-idf) or chi square. In terms of classification, I have 100 document related to computer science, 100 related to biology, and 100 that are associated with other disciplines. I would like to build a system that, given a new document, can determine which of those groups it belongs in.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Bag of words and tf-idf are some of the basic feature representations one could start with. When used with a simple linear model which consumes those features and spits out the group(s) to which an input document belongs you will have a decent benchmark model to beat.
Depening on which software you are using your mileage can vary.
For example: Python appears to have a better text processing capability than R ( my opinion only, based on experience - no sources to quote). For example: There is another feature representation called the Hashing Trick - essentially you take a word ( any word) from the input document and hash-map it to a fixed sized feature vector.
For example: If you think the dimensionaility of your problem would not be beyond 1024 words [in effect setting up 1024 binary/count variables]. The hashing function takes hashFunc('cat') =1023[meaning you will set the 1023rd column to 1/increment it by 1 if using a counts representation]; hashFunc('dog')=23[meaning you will set the 23rd column to 1/increment it by 1];hashFunc('cow')=957 and so on... This has the effect of dimensionaility reduction and is much more memory efficient. I think there are functions in Python which do this for you with relative easy ( search for Python hashingVectorizer) including the usual text preprocessing.
Another software which uses the hashing trick is the Vowpal Wabbit
Other techniques worth considering are predominently in the "how to reduce the dimensionality of your problem" domain and include things such as Singular Value Decomposition, Random Projection both applied on either the bag of words or tf-idf representation etc...
More advanced techniques will invole Deep Learning (neural net)
$\begingroup$
$\endgroup$
To my knowledge, most of the approaches for selecting feature modeling techniques are "rule of thumb"-type recommendations, based on the type of text classification problem you are working on, and the domain from which the documents arise. In my opinion, the most sound approach for selecting the feature modeling component of your pipeline is to have a hold-out development data set that you can use to evaluate the various approaches you are considering. Once you have made your selection, you can use the hold-out evaluation data set to determine the extent to which your results are likely to generalize to the larger population of document that you're looking to classify.