Suppose you have two datasets D1 and D2. Both are sampled from the same underlying distribution X. I want to use them to train a neural network. The features are all unsigned integers in range [0; 2^64].
Due to the fact that the features are on vastly different scales, I decided to use z-score normalization in conjunction with a sigmoid function. That means that I feed the z-score normalized data to the logistic function to map the features to the [0; 1] range.
At this point I am not sure at which point to normalize the data.
1.) I use D1 and normalize it with mean_1 and std_dev_1, which are obtained by only considering D1. I repeat the process for D2 and normalize it by using mean_2 and std_dev_2. Then I train the network with the two datasets sequentially.
2.) I add D1 and D2 to get a set D3, and normalize it by calculating mean_3 and std_dev_3 over the whole dataset (D1 + D2). Then I train the network with it.
2 Questions here:
a) Do the two methods lead to similar results? It is especially important to me as D2 may become available later to me than D1 and I have to know if I must retrain the network with the whole dataset.
b) When doing inference with the trained network, which parameters do I have to use to normalize the new inputs? So do I have to use mean_3 and std_dev_3 for example?
EDIT: I found out that mean and standard deviation of the combination of the two datasets can be calculated from mean and standard deviation of the original ones. That means (in theory) they could be trained sequentially and their distribution parameters could be combined to normate the inputs for inference.