Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Before performing the task of text mining, we need to select the features for characterizing each given document. Are there any systematic guidance on choosing the document features? How does the length of the document affect the feature selection process for the documents?

share|improve this question
what for ? Clustering ? Classification ? Related:… – steffen Sep 5 '12 at 6:46

There are a lot of ways for doing feature selection. I'll give you my "system" that works in my domain (insurance).

The first thing I usually try is calculating TF-IDF and taking the top 5-20% depending on how many features there are. To be honest, I don't worry too much about document length at this stage (or more specifically - the difference in lengths between documents) - I feel the TF portion of TF-IDF accounts for that, and IDF essentially creates a "corpus specific" stopword list. But I'm also not comparing Tweets to say War and Peace either.

I then take these features and use Naive Bayes to classify - I consider this to be my "baseline", but it's often good enough - especially since I frequently only need to quantify text for combination with more traditional structured data.

Hope this enough to get you started.

share|improve this answer

It really depends on the data and the problem you are trying to solve. Tf-Idf is very common but one might also consider if a binary weighting scheme is better.

What algorithm to choose for your particular task can be difficult to say ad-hoc but SVM's are very popular. The text mining project i did so far was with tf-idf score and linear support vector machine, with no feature selection.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.