Standardization in neural network online training

Question

It is common knowledge that the inputs to a neural network should be standardized to have mean 0 and variance 1 (see this thread for example, or the LeCun paper). And as long as one does batch training, there is no problem in applying this rule.

Consider however the case where one performs online learning (a.k.a stochastic learning), in which data samples come in successively. For that, one could use the Welford algorithm to calculate mean and variance on the fly and apply it directly to standardize the input. This could happen in two (only slightly different) ways:

1. either the current input is used to update the mean and variance and standardized thereafter with the updated values.
2. or the input is used to update the mean and variance, but for standardization the old values are used.

In both variants the goal is to train the network with correctly standardized inputs.

---

As simple this idea is, I could not find a reference after a short internet search. However, I would guess that this algorithm works well in practice and that it is used by many implementations. So the first two questions:

• Does anybody have references on or experiences with this online algorithm?
• Are there other algorithms used for standardization/normalization in the online-learning case?

One particular point that prevents me from simply applying the algorithms above is the extreme case of only a single data sample which is fed in repeatedly into the neural network. For this scenario, standardization will lead to a zero input at least in the second iteration. As a consequence, the neural network won't train anything.

• Is that a counter-argument against applying the above online algorithm? Can this problem be cured in a reasonable way (--somehow not all too brute-force-wise)?

Answer 1

Assuming you have an online system that runs every day from 8 AM to 5 PM, I would normalize based on the training data (whatever you are fitting on, e.g. last N days' data), and use the same mean and std dev values for the rest of the day/period. You can then re-run this calibration step for the next calendar day rolling the N day interval forward.

The worst that will happen is, when you encounter a value that is outside the range covered by your training data, it will have a very high or low z score, indicating it is an extreme example. This, in practice, is not a huge concern if you have sufficient training samples and assume a stationary process (which does not fluctuate massively, let's say, from day to day).

However if you update the mean and std deviations (i.e. your variable distribution) on the go, then the z-scores of observations at different stages in the day will have been generated from different distributions. So a z-score of 0.05 at iteration 1 is no longer comparable to a z-score of 0.10 at iteration 1000 since there is the danger that the distribution has drifted.

So what I suggest is learning your scaler from the training set and stick to it::

from sklearn.preprocessing import StandardScaler
def preprocess_data(X, scaler=None):
   if not scaler:
       scaler = StandardScaler()
       scaler.fit(X)
   X = scaler.transform(X)
   return X, scaler
X, scaler = preprocess_data(X_train) # fit scaler based on train set
X_test, _ = preprocess_data(X_test, scaler) # normalise using the train set's scaler
model.fit(X,y_train) # fit on normalised
model.predict_proba(X_test) # use the scaler (mean / var you have used from your training set)