I have read this article which compares various machine learning algorithms on SMS spam detection. I have been able to reproduce results for Naive Bayes and now i would like to try Random Forest as well.
With Naive Bayes, it's straightforward. The bag-of-words model is used and missing probabilities are handled with Laplace smoothing. Only words present in an SMS are tested against SPAM|HAM classes. However, i am not sure how to handle the input with random forest.
Suppose i have following setup:
- 5000 distinct words in training set, after stemming and removal of stop words
- text to classify is short, e.g. 10 words in average
- CART used as a tree model
- random forest selects subset of features, say 2*sqrt(5000) = 141 words for each split
- word frequency is used as feature value(could be also TF-IDF)
So my questions are:
- Generally speaking, regardless of article, can random forest be used effectively for short text classification when the feature space is large ? It seems to me that there may be a lot of weak classifiers due to large feature space and only handful of features present in data to classify.
- How to handle unseen words with random forest(not present in training set)? Should they be simply stripped from input or some technique similar to Laplace smoothing should be used ?
- If somebody perhaps read this short article, maybe could explain how author represented features for random forest ?