I am classifying companies into two classes ( a particular business type, or not that business type ), using a Naive Bayes Classifier. Specifically, I'm using PHP and PHP NLP Tools.
I have two reasonably accurate lists of companies to use as input for training and test sets ( business type, and not the business type ). The accuracy of these two lists has been validated internally, and by our customer who we are doing the work for.
I generate a document for each company using various features from a HUGE database of potential features describing these businesses ( industries they work in, certifications they have, products and services they sell, etc.. ).
I am mostly guessing at what features to throw at the system, and not seeing a lot of improvement when I make changes. I've read a lot about programmatically ( statistically ) selecting features that are (in laymen's terms) very significant to my "business type" companies, but not important to the non business type companies.
I'm curious what methods exist for this type of feature selection/optimization. For instance ... the entire set has roughly 25k different products and services. I recognize issues with doing this, and can see how selecting more 'relevant' features to my good set would be helpful.
The most simple methods I can try first would be very helpful.