Trees of ensembles. I have a large dataset (100k+), and it's growing everyday. I want to train it to predict a value (a regression problem). 
I've been finding that ensemble trees work the best for now, but in the future can imagine that it would be impossible to train the entire dataset. Scikit-learn says use SGDRegressor because you can partial fit it.
Would it ever make sense to split my dataset to several subsets based on some category and just tackle each subsets individually by creating a completely separate model for each category?  This way I can keep my subsets small and continue to use trees. 
Or should I just switch to SGDRegrssor and hope that imperfections of the model would be compensated by the large amounts of data. 
Thanks!
 A: Yes, making models that target particular subsets of your data makes a lot of sense, and you if you take that step you should take careful note of how the sub-models differ from the original 'model of everything'.
Although you haven't really given any details of your data to help answer the question, it is easy to think of situations where subsets of populations could exhibit differing relationships between explanatory variables and the target - to the point where the direction of the relationship could even be reversed. Increasing the size of the data set won't help with difficulties of this sort.
You mentioned that you have a large dataset - in general I would say that there are only reasons to not make individual models for the categorised data. One is that the results (e.g. which variables are important, the parameters for the variables) between the models don't differ. But you won't know that until you look. The other is that there isn't enough to data to support it but you don't seem to have that problem. 
