I have a dataset of ~43000 8k-filings (ad-hoc announcements from companies listed on the stock exchange) and i am trying to use decision trees with gradient boosting to develop a text classification algorithm (3 classes: positive, negative and neutral abnormal returns).
So far i removed stopwords from my documents and stemmed the remaining words. Still, if I count unique 1grams and 2grams of all documents I get about 130000 total unique 1grams and 1450871 unique bigrams total. I fear that without further feature reduction I will end up with curse of dimensionality.
Now I looked at the distribution of number of words per document and I find that only 356 documents have more than 2000 words and at the same time without these 356 documents my feature set would be reduced to only 89000 features.
So the question that comes to my mind is the following: Would it be ok to handle these documents with more than 2000 words as outliers and exclude them from my sample?
My further proceeding would be to use Tf-Idf, LSI and probably remove features with very low variance, then feed this to my algorithm.