9
$\begingroup$

I'm doing data preprocessing and going to build a Convonets on my data after.

My question is: Say I have a total data sets with 100 images, I was calculating mean for each one of the 100 images and then subtract it from each of the images, then split this into train and validation set, and I do the same steps to process on a given test set, but it seems like this is not a correct way doing it according to this link:http://cs231n.github.io/neural-networks-2/#datapre

"Common pitfall. An important point to make about the preprocessing is that any preprocessing statistics (e.g. the data mean) must only be computed on the training data, and then applied to the validation / test data. E.g. computing the mean and subtracting it from every image across the entire dataset and then splitting the data into train/val/test splits would be a mistake. Instead, the mean must be computed only over the training data and then subtracted equally from all splits (train/val/test)."

I'm guessing what the author is saying is that, do not compute mean and subtract it within each image but compute the mean of the total image set(i.e. (image1 + ... + image100)/100) and subtract the mean to each of the image.

I don't quite understand can anyone explain? and also possibly explain why what I was doing is wrong(if it is wrong indeed).

|cite|improve this question
$\endgroup$
  • $\begingroup$ Welcome to community. Please see my answer below. $\endgroup$ – usεr11852 Dec 8 '15 at 2:24
5
$\begingroup$

Let's assume you have 100 images in total; 90 are training data and 10 are test data.

The authors correctly asserts that using the whole 100 image sample to compute the sample mean $\hat{\mu}$ is wrong. That is because in this case you would have information leakage. Information from your "out-of-sample" elements would be move to your training set. In particular for the estimation of $\hat{\mu}$ , if you use 100 instead of 90 images you allow your training set to have a more informed mean than it should have too. As a result your training error would be potentially lower than it should be.

The estimated $\hat{\mu}$ is common throughout the training/validation/testing procedure. The same $\hat{\mu}$ is to be use to centre all your data. (I mention this later because I have the slight impression you use the mean of each separate image to centre that image.)

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the answer, that makes sense. And yes, I am calculating the mean for each images, so the estimated $\hat{\mu}$ should actually be calculated by the whole 90 training images instead of on each single image? Any reason why? Can't I centring each of the individual image in the 90 training set? $\endgroup$ – Sam Dec 8 '15 at 3:35
  • $\begingroup$ The estimate $\hat{\mu}$ should be an image itself. If you centre each individually the centring you do does not control for any overall trend in the whole sample. $\endgroup$ – usεr11852 Dec 8 '15 at 5:58
  • $\begingroup$ @usεr11852 Why would having more informed mean harm our model? This would not cause the information of "out-of-sample response variable",in any way interfere in our training, right? So why would the train error be low? $\endgroup$ – GeneX Nov 3 '17 at 13:56
  • $\begingroup$ Valid information will never "harm a model"; after deciding which model to use (based on some resampling/hold-out scheme) we will train the final model on all data. Nevertheless when training if we estimate $\hat{\mu}$ using the whole dataset, this additional information will be reflecting unrealistic good insights. This can lead to lower test-errors than expected exactly because we know something about out tests data that we would not otherwise be able to use during training. (Example: Say we develop a physical activity model. Our data include teenagers, adults and elderly and (cont.) $\endgroup$ – usεr11852 Nov 3 '17 at 15:11
  • $\begingroup$ ... by some fluke, all the elderly people end up in the test-set. If we calculate the mean-age in our training-set only, we will obviously get a lower mean-age than what our whole sample has. Using this obviously biased mean-age will probably deteriorate the model's A performance if A does not generalise well to different ages. If we calculate the mean-age in the whole dataset we will get a more representative mean-age. If we now use this unbiased mean-age in the model A we will probably get better performance than before despite A not generalising well to different ages.) $\endgroup$ – usεr11852 Nov 3 '17 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.