I have a training data set with both numerical and categorical variables, and one class variable. I want to build a classification model (e.g.,SVM), and for this goal I need to transform all variables into convenient format. I m confused about my categorical variables. Let me give you an example about one of them.

The categorical variable in each observation represents a Google search query (usually 3-10 comma-separated words, see example below).

search_id | query_words (categorical)  |..(other variables)| class variable
1         | how,to,grow,tree           |..                 | 4
2         | smartfone,htc,buy,price    |..                 | 7
3         | buy,house,realty,london    |..                 | 6
4         | where,to,go,weekend,cinema |..                 | 4
...       | ...                        |..                 | ...

The words in this categorical variable are disordered and the same words may occur in different observations (that's logical). Number of unique words for all observations = few thousands. Number of observations: ~150.000.000

Since this categorical variable (query_words) is very important for my classification analysis, I need to train my model with it. My question is how to represent it to use for e.g., SVM.

In each observation I can sort words alphabetically to order them. If I will use a numeric vector with few thousands elements (one per each unique word) I can represent this variable for each observation as e.g.:

query_words[1] = (0,0,..1,..0,..1,..1,..0,...1,..0) # very big vector

But I don't believe it will work effectively. How should I handle this categorical variable. I m using R for analysis.

  • 2
    $\begingroup$ What you asking is a pretty well-studied problem in machine learning, I would suggest you to google "svm, text classification". $\endgroup$ – Leo Mar 27 '12 at 21:38

You essentially have a feature (aka attribute) for each word. In many SVM implementations this is not as inefficient as you might think as they are generally optimized to handle sparse matrices.

This is very standard in text classification, usually not with a binary value for each attribute, but rather a calculation representing the words frequency in the document using Term Frequency (TF) and Inverse Document Frequency (IDF). Much of the research surrounds what that calculation should be. Since you seem to be dealing with queries and repetition of words in the query isn't a problem, you could use a binary value to start and see what kind of results you get from that.

  • $\begingroup$ Thank you, karenu, Did I understand you right, you think that having one of variables as a vector with few thousdand elements will be ok for SVM? I usually use TF-IDF for clustering text, don't think I can use tf-idf in this case. -Alex $\endgroup$ – user10243 Mar 30 '12 at 13:29
  • $\begingroup$ Yes, I haven't done a lot of text classification, but I've used SVM for image classification where each image is just represented by a feature for each pixel, making each instance several thousand features and its worked great. $\endgroup$ – karenu Apr 2 '12 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.