Feature selection for the text mining?

Question

Before performing the task of text mining, we need to select the features for characterizing each given document. Are there any systematic guidance on choosing the document features? How does the length of the document affect the feature selection process for the documents?

Answer 1

There are a lot of ways for doing feature selection. I'll give you my "system" that works in my domain (insurance).

The first thing I usually try is calculating TF-IDF and taking the top 5-20% depending on how many features there are. To be honest, I don't worry too much about document length at this stage (or more specifically - the difference in lengths between documents) - I feel the TF portion of TF-IDF accounts for that, and IDF essentially creates a "corpus specific" stopword list. But I'm also not comparing Tweets to say War and Peace either.

I then take these features and use Naive Bayes to classify - I consider this to be my "baseline", but it's often good enough - especially since I frequently only need to quantify text for combination with more traditional structured data.

Hope this enough to get you started.

Answer 2

It really depends on the data and the problem you are trying to solve. Tf-Idf is very common but one might also consider if a binary weighting scheme is better.

What algorithm to choose for your particular task can be difficult to say ad-hoc but SVM's are very popular. The text mining project i did so far was with tf-idf score and linear support vector machine, with no feature selection.

Answer 3

Approaches to feature selection have been developed since the 1960-es. It's a tricky problem to approach when looking for the optimal feature subset to select.

In the 1968-paper, Huges demonstrated that the performance of a classifier can peak and thereafter decline - on an independent test set - when adding still more features [G.F.Hughes. On the mean accuracy of statistical pattern recognizers, IEEE Trans. Inf. Theory, 14 (55-63) 1968]. This test-performance peaking is counter intuitive as one would expect that adding even more features to a classifier should not decline its performance. Acknowledge that all 'good' features are retained when adding still more (possibly noisy) features. The larger the training set size is, in terms of observations or 'cases', the later will a possible 'peak' occur.

Peaking is synonymous with overfitting of your text classifier.

Your question: "How does the length of the document affect the feature selection process for the documents?" Text length, and therefore training set size certainly influences the location of a possible peak, when pruning redundant features. The larger the training set size the less risk of overgeneralization (and hence peaking).

Your question: "Are there any systematic guidance on choosing the document features?" I recommend that you dig into the subject 'feature selection' in the scientific literature, for example [M. Kudo, J. Sklansky, Comparison of algorithms that select features for pattern classifiers, Pattern Recognition, 33(1), 25-41, 2000]. One strategy that stands out for feature selection is floating search by Pudil. It's more work to implement than a simple wrapper, but 'floating search' does often yield superior results.

Answer 4

There's a python library for feature selection TextFeatureSelection. This library provides discriminatory power in the form of score for each word token, bigram, trigram etc.

Those who are aware of feature selection methods in machine learning, it is based on filter method and provides ML engineers required tools to improve the classification accuracy in their NLP and deep learning models. It has 4 methods namely Chi-square, Mutual information, Proportional difference and Information gain to help select words as features before being fed into machine learning classifiers.

from TextFeatureSelection import TextFeatureSelection

#Multiclass classification problem
input_doc_list=['i am very happy','i just had an awesome weekend','this is a very difficult terrain to trek. i wish i stayed back at home.','i just had lunch','Do you want chips?']
target=['Positive','Positive','Negative','Neutral','Neutral']
fsOBJ=TextFeatureSelection(target=target,input_doc_list=input_doc_list)
result_df=fsOBJ.getScore()
print(result_df)

#Binary classification
input_doc_list=['i am content with this location','i am having the time of my life','you cannot learn machine learning without linear algebra','i want to go to mars']
target=[1,1,0,1]
fsOBJ=TextFeatureSelection(target=target,input_doc_list=input_doc_list)
result_df=fsOBJ.getScore()
print(result_df)