What I mean is the following: Instead of processing all the training data at once and calculating a model or hypothesis, we process one data point at a time and update the model directly afterwards.

I have seen the terms "on-line (or online) learning" and "incremental learning" for this. Is there a subtle difference? Is one term used more frequently? Or does it depend on the research community?

The Bishop book ("Pattern Recognition and Machine Learning") uses the terms on-line learning and sequential learning as synonyms but does not mention incremental learning. Are they really synonyms or is there also a small difference?


Regarding the first two, online and incremental learning, some authors underline that

On-line has to discard a sample after learning (no memory) and unlike to incremental learning is not allowed to store it. Source: this paper


To be honest, I have only rarely seen the term "incremental learning" come up. However, I have seen the "incremental" modifier used more often to describe specific out of memory implementations of various algorithms (e.g. incremental SVM, incremental decision trees) fairly frequently.

It seems that use of "incremental" is used synonymously with "online" or "sequential."


Some of the information contained in this post requires additional references. Please edit to add citations to reliable sources that support the assertions made here. Unsourced material may be disputed or deleted.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.