I am designing a scikit learn classifier which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the categories but it generates classifiers in the order of 100s of GBs binary file with very poor accuracy. So I think that having one classifier for each category would be helpful and would also help me to fine tune features for each category thereby improving accuracy, but this means 5k+ classifiers for each of these categories. So how to handle this large data requirements and which incremental classifiers to use for this case , considering the fact that I will keep on getting additional training data as well as may discover new categories?

Update :

I ran the experiment on 1 lac samples and found that using boosted decision trees gives a accuracy of 65% on validation set which is better then all other classifiers I tried. So will increasing the training data help improve accuracy? I found that increasing the training data size incrementally going from 80k samples to 1 lac samples just provides an additional 2-3 % increase in accuracy. So will increasing training set size increase accuracy? and will using LSTMs and RNN further increase the accuracy ?

The number of features are about 120 which are mostly text based and most are categorical with text based values with large cardinality i.e many features may have huge number of possible values and available RAM IS 128gb with 12 core CPU.

  • $\begingroup$ What about nearest centroids ? You only need : number of classes * number of features to store them (even less if your data is sparse) and you can train them with streaming methods (you seem to have a large amount of data). I don't know how they are implemented in python though, but writing your own is really easy. $\endgroup$
    – RUser4512
    Sep 14, 2015 at 9:19
  • 1
    $\begingroup$ What is the dimensionality of the feature/input space? What methods have you already tried giving the large models with poor accuracy. $\endgroup$ Sep 14, 2015 at 9:29
  • $\begingroup$ You need to tell us more, detail about which methods did you try, nature of the variables you use to construct the classifyer, ... $\endgroup$ Sep 14, 2015 at 11:30
  • $\begingroup$ @kjetilbhalvorsen ya updated $\endgroup$
    – stats101
    Sep 14, 2015 at 12:20
  • $\begingroup$ I have no experience with such huge problems, but maybe you should try , with the "categorical vars with huge n of categories" to see if it is possible to collapse categories, to get a more managable number of categories? Maybe it is possible to use the lasso idea for this. If you first get some parameter estimates, then recodes the categories, after sorting them in the order of the preliminary estimates, and use a coding with "difference from adjacent category", then use lasso, you can see which of the adjacent category betas which are taken to zero, and then you can throw together those. or $\endgroup$ Sep 14, 2015 at 14:05

1 Answer 1


This is more of an extended comment as you have not given sufficient information to give detailed advice. Also, I have no experience with such a large-scale problem, and I suspect few really has. You say "I am designing a scikit learn classifier which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year." which is a HUGE problem, and probably a major research project. You should take time to look at some papers describing similar efforts, like http://vision.stanford.edu/documents/DengBergLiFei-Fei_ECCV2010.pdf which describes trying to classify millions of images into 1000+ categories. I will cite a few paragraphs to show the inmensity of the project:

In practice, all algorithms are parallelized on a computer cluster of 66 multicore machines, but it still takes weeks for a single run of all our experiments. Our experience demonstrates that computational issues need to be confronted at the outset of algorithm design when we move toward large scale image classification, otherwise even a baseline evaluation would be infeasible.

weeks, for a single run of one experiment, on a cluster of 66 machines

Do you have the resources for such a project?

If not, and even then, you should start out with some simplified project, see how that goes, and continue from that.

One idea: with thousands of categories, there must be some hierarchcal structure to the space of categories. If you can start mapping out that space, maybe organizing the categories in a binary tree, you could try a binary classifier for each level of the tree. Just a thought!

Another idea: mapping out the space of categories something like in multidimensional scaling .... would give coordinates to the categories, and then you could build a predictor for those coordinates. Something like that could work, or not, we do not know until somebody tries! I guess this is really white spots on the map ...

Good luck!

  • $\begingroup$ what about using LSTMS and neural networks? $\endgroup$
    – stats101
    Nov 13, 2015 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.