I currently read the paper Efficient Online and Batch Learning using Forward-Backward Splitting by John Duchi and Yoram Singer. I am very confused about the usage of the terms 'Online' and 'Batch'.
I thought 'Online' means we update the weight parameters after processing one unit of the training data. Then we use the new weight parameters to process the next unit of the training data.
However, in the paper above, the usage is not that clear.
To me it looks like they are using batch and online learning correctly. In section 3 they are working on the whole dataset to perform learning, i.e., batch learning, while in section 4 they switch to stochastic gradient following which can be used as an online learning algorithm.
I've never used stochastic gradient following as an online learning algorithm; however, it is possible to simply stop the optimization process in the middle of a learning run and it still being a useful model. For very big datasets this is useful since you can measure the convergence and quit learning early. You can use stochastic gradient following as an online learning method since you update the model for every new datapoint, as I think you yourself said. Although, I'd be careful about calling it "per training data." Training data is a dataset, not a datapoint, but I think I understood you since you said "per training data."
Batch Versus On-line Learning
The on-line and batch modes are slightly different, although both will perform well for parabolic performance surfaces. One major difference is that the batch algorithm keeps the system weights constant while computing the error associated with each sample in the input. Since the on-line version is constantly updating its weights, its error calculation (and thus gradient estimation) uses different weights for each input sample. This means that the two algorithms visit different sets of points during adaptation. However, they both converge to the same minimum.
Note that the number of weight updates of the two methods for the same number of data presentations is very different. The on-line method (LMS) does an update each sample, while batch does an update each epoch, that is,
LMS updates = (batch updates) x (# of samples in training set).
The batch algorithm is also slightly more efficient in terms of number of computations.
In short,
Online: Learning based on each pattern as it is observed.
Batch: Learning over groups of patters. Most algorithms are batch.
Source: http://machinelearningmastery.com/basic-concepts-in-machine-learning/
