Nonlinear Dynamic Online Classification: Looking for an Algorithm I have two predictors a,b that I want to use combine to classify data. a is stable, it will always produce the same prediction for the same input. b will change and probably improve in time (because it represents user feedback).
I am looking for an algorithm  to combine a and b that is nonlinear and trained step by step (whenever new user feedback comes in, thus b changes) and that weights more recent events higher than those in the past, because earlier events represent an older version of b.
 A: Your problem seems to be well suited for online learning.
You can use stochastic or mini-batch gradient descent to train a neural network continuously over time.
In stochastic gradient descent, you take one gradient descent step each time you get a new training example. On mini-batch, you do it each time you gather a batch of n training examples.
Since your local minima changes over time, your neural net will continuously adapt its weights as newer data comes in, fitting the new local minimum.
You can also play around with the step size. The larger it is, the more it will effectively weight more recent data (e.g. the easier to escape from the older local minimum) but could also become super unstable. Worth tuning as a hyperparameter and running backtests in time.
You can watch this video by Andrew Ng for an example of online learning: https://www.youtube.com/watch?v=dnCzy_XKGbA
A: If you want it to be a non-linear model, how about a simple neural network with two input neurons, and the output layer having the number of labels possible (or if you are using a regression model, use a sigmoid neuron for output and scale it to your preference). With one hidden layer containing 3-4 sigmoid/logistic neurons, and using back-propagation to improve with time. The model I described satisfies your requirements. 
hope this helps.  
