It is common knowledge that the inputs to a neural network should be standardized to have mean 0 and variance 1 (see this thread for example, or the LeCun paper). And as long as one does batch training, there is no problem in applying this rule.
Consider however the case where one performs online learning (a.k.a stochastic learning), in which data samples come in successively. For that, one could use the Welford algorithm to calculate mean and variance on the fly and apply it directly to standardize the input. This could happen in two (only slightly different) ways:
- either the current input is used to update the mean and variance and standardized thereafter with the updated values.
- or the input is used to update the mean and variance, but for standardization the old values are used.
In both variants the goal is to train the network with correctly standardized inputs.
As simple this idea is, I could not find a reference after a short internet search. However, I would guess that this algorithm works well in practice and that it is used by many implementations. So the first two questions:
- Does anybody have references on or experiences with this online algorithm?
- Are there other algorithms used for standardization/normalization in the online-learning case?
One particular point that prevents me from simply applying the algorithms above is the extreme case of only a single data sample which is fed in repeatedly into the neural network. For this scenario, standardization will lead to a zero input at least in the second iteration. As a consequence, the neural network won't train anything.
- Is that a counter-argument against applying the above online algorithm? Can this problem be cured in a reasonable way (--somehow not all too brute-force-wise)?