# Simple text classifier: classification taking forever?

I work for a small tech startup, and I want to classify our users into demographics based on the domain of their email address. When users sign up to our site, they can enter a job category or pick "other". The goal is to classify as many of the "other" type as possible using a bag-of-words approach.

To do this, I have written some code in Python. For each user, I look at the domain name of their email address and scrape the text from their homepage (using Beautiful Soup). I also look for an "About Us" page, which I also scrape. What I'm left with is a map of domains to text. Some domains are classified (i.e., users whose email address comes from this domain have self-classified their job types), and some aren't (those users who have self-classified as "other"). The total data set for classified users is about 2000 (neglecting domains like gmail and hotmail and [I can't believe I'm about to type this] aol). I'm using a train/test split of 75/25.

Using scikit-learn, I'm trying to implement a simple classifier, but there seems to be an issue with either convergence or performance. The data set doesn't seem particularly big, but the two classifiers I've tried (Perceptron and RidgeClassifier) seem to be having some issues finding a fit. I haven't really tried to change the parameters for the classifiers, and it's not clear to me which nobs I should be turning.

I lack intuition into this problem, and it's difficult for me to tell whether the issues I'm having are due to not enough data, or what. I'd like to know

• Am I barking up the wrong tree? Has anyone tried something like this and made it work?
• Do other ML packages for Python do a better job of text classification? (I'm looking at you, nltk.)
• Is my data set large enough? Are there any "rules-of-thumb" for how much data you'd need for something like this (~5-10 categories)?
• What's a reasonable amount of time for the learning to take? Are there any hints that will tell me the difference between "this is really hard" and "this isn't going to work"?
• I've tried to follow the examples here and here. These examples are pretty speedy, so it makes me worry that I don't have enough data to make things work nicely. Is the "20 newsgroups" classification problem typical, or does it show up because it's easily solvable?

Any guidance here would be appreciated!

As an update: the huge performance hit seems to come from the "vectorizer": that is, the thing that maps a vector of words to the reals. For some reason, Tfidf was taking a long time to do its thing---I switched to a different vectorizer, and now things run quite quickly.

In regards to the actual learning, I've found that the Naive Bayes routines work pretty well out of the box (f-score around 70-75%, which is good enough for now). The model that I found works the best, however, is one based on a linear SVM (scikits.svm.LinearSVC), which gets me somewhere in the 80-85% range with a bit of tinkering.

• Can you be more specific about what issues your classifiers appear to be having? Do they perform above chance? – jerad Feb 25 '13 at 8:16
• @jerad: This is my first real machine learning exercize, and I just feel like I can't distinguish between "this problem is taking a long time to solve" and "the fact that this is taking so long is a sign of a bigger problem". So, as a first pass, I was asking about performance---what should I expect from a text classification algorithm? The samples I have are cooked up things from the scikits website that run very quickly, and it's not clear to me whether these examples are representative, or if they are just simple textbook exercizes that work well. – BenDundee Feb 25 '13 at 14:42
• I'd agree with @Peter's suggestion below - Naive Bayes looks like an obvious first approach to try in your setting - it's designed for bag-of-words data and generally it works pretty well on text data. If it doesn't give good enough results then you'll want to try something smarter but that will require digging deeper into the problem and, for better results than NB, you'll need to spend some time tuning your classifier which usually brings in quite a bit of time overhead. – Bob Durrant Feb 27 '13 at 10:56