I do natural language processing and use words as features. But in online learning, the new training data will exist new words which does not appear in the previous data. How to deal with these new words? how to add new features in Stochastic Gradient Descent model in online learning?
I have been facing a similar problem for a while. I came up with the following strategies:
- For large featurespaces such as with text data, you would ideally need to perform some kind of feature reduction. If you separate this step from the actual model learning step, and perform it once a month or so, you can see whether the new features are relevant enough to require updating the model. It is possible that while you get a lot of new words in that one month, they are not significant enough to warrant updating the (reduced) featurespace of your model.
- You can set a threshold of word occurrences, and only when the number of occurrences of the new words cross that threshold, do you consider updating your model. This threshold can be learned using existing data on a cross-validation set.
- Perform some kind of time-based attentuation. Perhaps old data samples are not that important, especially if you trying to model preferences, which can change over time.
- Unfortunately, at the end, I decided that for my problem, I would have to retrain the model after using all the above steps. So first keep a count of new feature occurrences, once they cross a threshold, perform feature reduction to test significance of the new features, and then re-learn my model using the $n$ most recent data points, discarding the ones that are "too old".
You can also look into kernel-based methods that deal with infinite featurespaces, essentially allowing the dimensions to grow indefinitely.
One approach is to use feature hashing, where rather than using the words directly you hash them into some fixed size feature space. When you encounter a new word, you apply a hash function, and mod the result so it fits into your feature space. Of course some words will end up being assigned to the same feature, but in practice it usually doesn't hurt performance too much.