1
$\begingroup$

I have a simple question. I am training a neural network feeding it with normalized data patterns using Gaussian normalization. My question arises when I see that some people use the mean and standard deviation of the training data to normalize the training data (which is logical) and the mean and standard deviation of the validation data to normalize the validation data (which is not so logical).

The validation data is supposed to be used to test the accuracy of the network in a real application, but in many applications you do not know beforehand what data you are going to predict/classify. For example: if you trained your NN to approximate a mathematical function "$f$" such that $y=f(x)$ and you want to know the output for a given value $x=i$, you only have that value and it cannot be normalized without other data.

So, I think that the $x=i$ value should be normalized using the mean and sd values of the training data, which makes more sense to me. If you were using [Max- min] standardization you would use the maximum and minimum values of your training data as well.

What do you think?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
0
$\begingroup$

There's a situation that using the mean and std from the test data might make sense.

Say I want to predict whether a student gets a scholarship or not based on his/her test results.

If my training data are students from school A, my test data are students from school B, and A and B use two different grading systems.

Then using the mean and std of school B would be better.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Yes, that situation has a point, since the training and the test group have different mean and SD values. But the data I am trying to predict belongs to the same group. This question arised looking at the code of James McCaffrey for the well-known Iris classification problem. He normalizes both training and validation data separately, but imagine I have now a single flower and I want the network to classify it. I would have to normalize using the training data mean and SD values. $\endgroup$
    – Cándido Otero
    May 8, 2016 at 11:00
  • $\begingroup$ @CándidoOtero yes then I would agree that training data mean and SD values are more appropriate. $\endgroup$
    – dontloo
    May 8, 2016 at 13:02
  • $\begingroup$ The widely used definition of test data , is a dataset that you don't have access to, when building a model. If a model is trained with summary statistics from a dataset, calling that a test dataset is a stretch. $\endgroup$
    – shark8me
    May 14, 2016 at 3:18
  • $\begingroup$ @shark8me yes but I would think of the normalization step as preprocessing instead of training. $\endgroup$
    – dontloo
    May 14, 2016 at 3:27
  • $\begingroup$ My guess is that the model will not generalize, since it has not seen data normalized in this fashion. In the situation where training and test datasets are from different distributions, the model cannot bridge the gap. $\endgroup$
    – shark8me
    May 14, 2016 at 3:31
2
$\begingroup$

Training mean and SD is used by Validation data on an assumption that your validation set is a representative of training set.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Right, and assuming that it is a representative sample of the entire available data, both mean and SD values are close from the training data values. The problem is still that when you implement the neural network predicting single values you are forced to use training data mean and SD values. $\endgroup$
    – Cándido Otero
    May 8, 2016 at 10:43
1
$\begingroup$

Using the mean and standard deviation of the validation set is a form of leakage, where the model gets to know about the attributes of the validation/test set.

Therefore your idea of using the mean and standard deviation for normalizing both train and test datasets, is the right approach.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.