I am designing a scikit learn classifier which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the categories but it generates classifiers in the order of 100s of GBs binary file with very poor accuracy. So I think that having one classifier for each category would be helpful and would also help me to fine tune features for each category thereby improving accuracy, but this means 5k+ classifiers for each of these categories. So how to handle this large data requirements and which incremental classifiers to use for this case , considering the fact that I will keep on getting additional training data as well as may discover new categories?
I ran the experiment on 1 lac samples and found that using boosted decision trees gives a accuracy of 65% on validation set which is better then all other classifiers I tried. So will increasing the training data help improve accuracy? I found that increasing the training data size incrementally going from 80k samples to 1 lac samples just provides an additional 2-3 % increase in accuracy. So will increasing training set size increase accuracy? and will using LSTMs and RNN further increase the accuracy ?
The number of features are about 120 which are mostly text based and most are categorical with text based values with large cardinality i.e many features may have huge number of possible values and available RAM IS 128gb with 12 core CPU.