I am using Vowpal Wabbit 7.10.0 (VW) to learn and predict categories on text data. However, my text data for each record is not like an article or another decent-size text document, but rather a couple of sentences, like a title and subtitle and keywords.
I have around 10,000 labeled records I can use for validation, training, and testing, and around 1-2 millions unlabeled records. Its a multi-class problem with around 100 class labels, also imbalanced.
What would be the best pre-processing and input format to get the most of such data with VW?
My experience tells me that VW models should be sensitive to class imbalance problem. Here is another source that confirms it. Is that right?
As for choosing a model, I decided that I would rather take into account word combinations through n-grams then discover latent variables based on frequency counts (because texts are too short.) Besides, some texts tend to list a word for 100s of times (for SEO) in my data. Hence, I don't go TF-IDF. Is that right or not? I guess, I can combine both n-grams and bag-of-words as different namespaces. But what classifier with what params to start with?
So far I tried it in three different ways of data pre-processing: (1) unprocessed text with only punctuation removed, (2) tokenization, lemmatization (not stemming), removed stopwords, (3) in addition to (2), bag of words, i.e. word:word_count format.
The results are not satisfactory with a very basic setting (this example used 16 classes, not 100):
vw input.vw -c -k --passes 300 -b 24 --ect 16 -f model.vw vw input.vw -t -i model.vw -p preds.txt
Error rate is about 0.68 even on the training set.
I have some time limits to explore deeply all kind of setting, and really need quick and informative advise: what is the best pre-processing technique in my case, and what model implemented in the latest VW should I use. These two issues are related.