48
$\begingroup$
  1. Do I transform all my data or folds (if CV is applied) at the same time? e.g.

    (allData - mean(allData)) / sd(allData)

  2. Do I transform trainset and testset separately? e.g.

    (trainData - mean(trainData)) / sd(trainData)

    (testData - mean(testData)) / sd(testData)

  3. Or do I transform trainset and use calculations on the testset? e.g.

    (trainData - mean(trainData)) / sd(trainData)

    (testData - mean(trainData)) / sd(trainData)

I believe 3 is the right way. If 3 is correct do I have to worry about the mean not being 0 or the range not being between [0; 1] or [-1; 1] (normalization) of the testset?

$\endgroup$
43
$\begingroup$

The third way is correct. Exactly why is covered in wonderful detail in The Elements of Statistical Learning, see the section "The Wrong and Right Way to Do Cross-validation", and also in the final chapter of Learning From Data, in the stock market example.

Essentially, procedures 1 and 2 leak information about either the response, or from the future, from your hold out data set into the training, or evaluation, of your model. This can cause considerable optimism bias in your model evaluation.

The idea in model validation is to mimic the situation you would be in when your model is making production decisions, when you do not have access to the true response. The consequence is that you cannot use the response in the test set for anything except comparing to your predicted values.

Another way to approach it is to imagine that you only have access to one data point from your hold out at a time (a common situation for production models). Anything you cannot do under this assumption you should hold in great suspicion. Clearly, one thing you cannot do is aggregate over all new data-points past and future to normalize your production stream of data - so doing the same for model validation is invalid.

You don't have to worry about the mean of your test set being non-zero, that's a better situation to be in than biasing your hold out performance estimates. Though, of course, if the test is truly drawn from the same underlying distribution as your train (an essential assumption in statistical learning), said mean should come out as approximately zero.

$\endgroup$
  • $\begingroup$ Thats what I thought. Thank you for clarifying this! $\endgroup$ – DerTom Sep 30 '15 at 16:23
  • $\begingroup$ Clearly, one thing you cannot do is aggregate over all new data-points past and future to normalize your production stream of data. Why not? $\endgroup$ – Anmol Singh Jaggi May 7 '16 at 12:32
  • 1
    $\begingroup$ @AnmolSinghJaggi Its the "and future". If you haven't actually collected the data yet, you cannot normalize using it. $\endgroup$ – Matthew Drury May 7 '16 at 15:43
  • 5
    $\begingroup$ @floodking If you think of training data as "past" data, and testing data as "current or future", by aggregating across your test data you implicitly use information about the future of X. Data leakage is not just about leaking $y$ into your predictors, it is also about leaking information from the future. A good rule of thumb is that you should be able to make predictions using only one row or your testing data, otherwise you are using the future. $\endgroup$ – Matthew Drury Nov 1 '17 at 18:17
  • 1
    $\begingroup$ @MatthewDrury. Thanks for your clear explanation. I agree with you now. Only the third way is correct. $\endgroup$ – floodking Nov 2 '17 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.