Is setting a self learning system possible via incremental (online) learning? Self learning and incremental learning are all new to me. I am trying to develop a system for one of my case. Simply I have a data set (with about 90K observations and 400 features) for a binary classification problem. I first select the important features via DRF and model it with GBM. Once the model is deployed, new data (one data point) will be provided on a daily (or weekly) basis. I'd like to use this new information to improve my model without re-training it.
From my searches incremental (online) learning sounds suitable for my problem. However, the more I read the more I am confused. For example, in the related scikit-learn's page, it says incremental learning is used (via partial_fit) for very large data-sets when it is not possible to fit them in the memory.
Would it be suitable to use incremental learning for my problem, what are your suggestions or warnings? Is there something similar on H2O?
Could you suggest a proper method for my problem? Any tutorials would be highly appreciated.
Thanks in advance!
 A: There are two ways I'd approach this, neither of which is really incremental learning.
Because you have 90,000 initial data samples, and are only increasing by 1 per day, or even less frequent, what I would do is collect them, and every three months retrain the model on all the available data. I.e. After 3 months you'd have 90,090 data samples. You'd expect to be getting almost identical performance, as you only have 0.1% more data.
In your comment you say: "I cannot use the previous model, which uses a different feature set, as checkpoint". If you are not just adding more data samples, but more features (i.e. not just more rows, but more columns, too) then the above is still good, but you have to somehow fill the missing data in the previous 90K data samples.
The other alternative I would consider is to use model ensembles. There are many ways we could approach this, but the simplest would be to still train on all available data. Perhaps start with three models, m1, m2, m3. Then once you have a worthwhile batch of new data, train model m4 on the combination of previous and new data, and on all features. Now you throw m4 into the ensemble.  After another worthwhile batch of new data, create model m5 and so on.
Once you start to collect a lot of models, I might start measuring their standalone performance, and selecting the best N to use in the ensemble. (Or even better, working out which N models work best together.) Or if you use a stacked ensemble (built into H2O), theoretically at least it will be doing this for you. (However to use a stacked ensemble I think all models must have been built with the same number of features.)
Because there are so many approaches the key thing is to set up a proper test of your whole system. Start with, say, 89000 samples. Then add the next 100 samples, and evaluate. Then the next 100 and evaluate.
If you do want to go down the route of using checkpoints, then the key thing to remember, with H2O, is that in addition to have more training data, you must increase epochs (deep learning) or ntrees (random forest and gbm), or it will not do anything. E.g. if you trained the inital model, m1 with 90,000 training samples, and ntrees=100. Now you have 90,100 training samples, and want to retrain, you would set the checkpoint model to be m1, and increase ntrees to 110. Or to 200?  (That is why this is not my first choice: by increasing ntrees or epochs you might start over-fitting on the original 90,000 samples.)
Again, how much to increase ntrees by is a hyper-parameter, and you need to simulate the updates, to see what works best.
I'm fairly sure you cannot add new features, with checkpointing. So, to get that to work you'd have to prepare the columns in advance, filled with zeros, for instance.
