Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My question is straightforward: I want to do statistical classification in a database, mostly of floats, but one of the fields of the data is a big text. By big I mean up to 1000 words.

What is the best way to deal with that?

UPDATED: It's supposed to be supervised learning: I have some rows that have the values "Closed" and "Not Closed", and want to estimate this row for the other ones.

share|improve this question
Could you describe what do you mean by 'classification'? Usually classification is understood as a supervised learning task, while in your case you seem to talk about unsupervised learning (clustering)? – danas.zuokas Jun 26 '12 at 5:57
@danas.zuokas: updated the question! – Lucas Reis Jun 26 '12 at 12:17

You need to quantify text, that is to create additional variables from that text which would help classification. These might be the length of a text, the frequency of occurrence of certain words or phrases (like in spam filtering) and so on. It might even be the whole document-term matrix, but in this case you will have a lot of non informative variables so you need to think of some kind of regularization.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.