When to normalize data when using two datasets from the same distribution? Suppose you have two datasets D1 and D2. Both are sampled from the same underlying distribution X. I want to use them to train a neural network. The features are all unsigned integers in range [0; 2^64].
Due to the fact that the features are on vastly different scales, I decided to use z-score normalization in conjunction with a sigmoid function. That means that I feed the z-score normalized data to the logistic function to map the features to the [0; 1] range.
At this point I am not sure at which point to normalize the data.
1.) I use D1 and normalize it with mean_1 and std_dev_1, which are obtained by only considering D1. I repeat the process for D2 and normalize it by using mean_2 and std_dev_2. Then I train the network with the two datasets sequentially.
2.) I add D1 and D2 to get a set D3, and normalize it by calculating mean_3 and std_dev_3 over the whole dataset (D1 + D2). Then I train the network with it.
2 Questions here: 
a) Do the two methods lead to similar results? It is especially important to me as D2 may become available later to me than D1 and I have to know if I must retrain the network with the whole dataset.
b) When doing inference with the trained network, which parameters do I have to use to normalize the new inputs? So do I have to use mean_3 and std_dev_3 for example?
EDIT: I found out that mean and standard deviation of the combination of the two datasets can be calculated from mean and standard deviation of the original ones. That means (in theory) they could be trained sequentially and their distribution parameters could be combined to normate the inputs for inference.
 A: You should apply the same transformation to all individuals.
Don't use method 1; it will be biased. An easy way to realize this is to imagine that two individuals with identical features exist in $D_1$ and $D_2$. You would want these two individuals to also be identical in the transformed datasets, but your method 1 doesn't allow this.
Method 2 is OK. If you want to train sequentially, another option would be to apply the transformation induced by mean_1 and std_dev_1 to all data points; note however that this can lead to issues if future data points are vastly different from the data in $D_1$.
A: If D1 and D2 are truly from the same distribution, then as long as D1 has at least a few hundred datapoints, you shouldn't be seeing much variation in the mean and standard deviation, so normalizing all the data based on D1 shouldn't pose much of a problem. Normalizing each subset of data based on its own mean and standard deviation means that your overfitting is going to follow more from the subset sample size than the overall sample size, and in a way you're introducing a spurious feature of "what subsample did this datapoint come from?". Normalizing the data with different means and std shouldn't affect the result of the neural net training so much as how long it takes to converge, and if you're worried about the latter, consider taking the results from one as the initial values for the next.  
A: A few remarks:


*

*If both datasets are from the same distribution, both procedures should give the same result (as D1 and D2 would have the same mean and variance). But apparently they are not.

*Check what are actual mean and variance for each dataset. If they are the same - it does not matter. If they are different, then normalize each dataset with its own mean and variance (method 1)

*Train on a shuffled dataset $D3 = \bar{D1} \cup \bar{D2}$, where a bar means normalization. Training first on one dataset and then on another may cause a lot of problem and should be done with caution (see catastrophic forgetting).

*If new inputs will be from the same source as D2, then normalize them with mean and variance of D2.

*Applying sigmoid on normalized values may not be necessary. 
