Suppose you have two datasets D1 and D2. Both are sampled from the same underlying distribution X. I want to use them to train a neural network. The features are all unsigned integers in range [0; 2^64].

Due to the fact that the features are on vastly different scales, I decided to use z-score normalization in conjunction with a sigmoid function. That means that I feed the z-score normalized data to the logistic function to map the features to the [0; 1] range.

At this point I am not sure at which point to normalize the data.

1.) I use D1 and normalize it with mean_1 and std_dev_1, which are obtained by only considering D1. I repeat the process for D2 and normalize it by using mean_2 and std_dev_2. Then I train the network with the two datasets sequentially.

2.) I add D1 and D2 to get a set D3, and normalize it by calculating mean_3 and std_dev_3 over the whole dataset (D1 + D2). Then I train the network with it.

2 Questions here:

a) Do the two methods lead to similar results? It is especially important to me as D2 may become available later to me than D1 and I have to know if I must retrain the network with the whole dataset.

b) When doing inference with the trained network, which parameters do I have to use to normalize the new inputs? So do I have to use mean_3 and std_dev_3 for example?

EDIT: I found out that mean and standard deviation of the combination of the two datasets can be calculated from mean and standard deviation of the original ones. That means (in theory) they could be trained sequentially and their distribution parameters could be combined to normate the inputs for inference.

  • $\begingroup$ Why this question has a bounty but still no satisfactory answer? Well, I think you have to provide more information about datasets D1 and D2. What is the source (some physical process?) and format (is this 1-d array of numbers?). Knowing goal (e.g. is it binary classification?) would also be helpful. $\endgroup$ – hans Aug 9 '18 at 15:01
  • $\begingroup$ @hans: the data is basically metadata on network traffic (e.g. average number of packets/sec). What I try to do is to find anomalies in the traffic. The idea was that in a perfectly fine network the normal traffic has some distinct pattern and that anomalies cause a deviation from this pattern. $\endgroup$ – DocDriven Aug 10 '18 at 9:28

You should apply the same transformation to all individuals.

Don't use method 1; it will be biased. An easy way to realize this is to imagine that two individuals with identical features exist in $D_1$ and $D_2$. You would want these two individuals to also be identical in the transformed datasets, but your method 1 doesn't allow this.

Method 2 is OK. If you want to train sequentially, another option would be to apply the transformation induced by mean_1 and std_dev_1 to all data points; note however that this can lead to issues if future data points are vastly different from the data in $D_1$.

  • $\begingroup$ '(...) imagine that two individuals with identical features exist in D1 and D2. You would want these two individuals to also be identical in the transformed datasets' - not exactly true. Quite often you do the opposite: (especially if your dataset is small) you augment data so one example is presented a few times slightly different for a NN. $\endgroup$ – hans Aug 9 '18 at 9:17
  • $\begingroup$ @hans: so what you're saying is that method 1 is viable? $\endgroup$ – DocDriven Aug 9 '18 at 9:26
  • $\begingroup$ @DocDriven I am saying that this argument cannot be used to reject method one. $\endgroup$ – hans Aug 9 '18 at 9:32

If D1 and D2 are truly from the same distribution, then as long as D1 has at least a few hundred datapoints, you shouldn't be seeing much variation in the mean and standard deviation, so normalizing all the data based on D1 shouldn't pose much of a problem. Normalizing each subset of data based on its own mean and standard deviation means that your overfitting is going to follow more from the subset sample size than the overall sample size, and in a way you're introducing a spurious feature of "what subsample did this datapoint come from?". Normalizing the data with different means and std shouldn't affect the result of the neural net training so much as how long it takes to converge, and if you're worried about the latter, consider taking the results from one as the initial values for the next.

  • $\begingroup$ I don't understand your last statement. Which initial values should I take for what? $\endgroup$ – DocDriven Aug 8 '18 at 14:17
  • $\begingroup$ @DocDriven When you initialize a neural net, you have to seed it with initial values for the weights. Seeding it with weights from previous training can speed up the retraining, although there are risks such as overfitting. $\endgroup$ – Acccumulation Aug 8 '18 at 14:33

A few remarks:

  1. If both datasets are from the same distribution, both procedures should give the same result (as D1 and D2 would have the same mean and variance). But apparently they are not.

  2. Check what are actual mean and variance for each dataset. If they are the same - it does not matter. If they are different, then normalize each dataset with its own mean and variance (method 1)

  3. Train on a shuffled dataset $D3 = \bar{D1} \cup \bar{D2}$, where a bar means normalization. Training first on one dataset and then on another may cause a lot of problem and should be done with caution (see catastrophic forgetting).

  4. If new inputs will be from the same source as D2, then normalize them with mean and variance of D2.

  5. Applying sigmoid on normalized values may not be necessary.

  • $\begingroup$ Thank you for clarifying this. Regarding point 1.) that the two datasets are from the same distribution: They are, but mean and variance do not always exactly equal the mean and variance of the population, e.g. there are slight differences. Does this mean that it does not matter with which parameters I normalize new inputs (point 4)? $\endgroup$ – DocDriven Aug 9 '18 at 10:06
  • $\begingroup$ @DocDriven You are right, there may be small differences. If datasets are big, the differences should be small and irrelevant. If datasets are small, then the main problem is size of datasets and how to augment it. How big are they precisely? $\endgroup$ – hans Aug 9 '18 at 11:00
  • $\begingroup$ around 1000 to 2000 samples per set. So I suppose the differences should be small. $\endgroup$ – DocDriven Aug 9 '18 at 11:23
  • 1
    $\begingroup$ Ok, and how are the sets different form each other? For example in CV domain, if I had two sets of pictures done with different cameras, I would normalize it separately. But if I had two set done with the same camera but on other days, I would merge them and normalize together. $\endgroup$ – hans Aug 9 '18 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.