1
$\begingroup$

I have used cross validation on my training set to choose the parameters that gives the lowest error for different machine learning models. I then used these models to predict the target values of my testing set and then calculated the error.

I tried normalizing the data and went through the same process as above, but I was surprised to see that my normalized cross validation error was worst than my not normalized cross validation error. However, my normalized testing error was better than my not normalized testing error.

Should I go with the normalized or not-normalized data?

$\endgroup$
3
  • $\begingroup$ What do you mean by "error"? $\endgroup$ Mar 29, 2018 at 12:34
  • 1
    $\begingroup$ @Frank Harrell RMSE,MAE,RSE $\endgroup$ Mar 29, 2018 at 13:04
  • 1
    $\begingroup$ What ML algorithms are you using? Different algorithms respond differently to normalization... $\endgroup$
    – jbowman
    Jan 13, 2023 at 3:13

2 Answers 2

0
$\begingroup$

Normalisation is mainly intended to create model stability by standardising value scales and to remove sources of absolute variability in favour of exploring relative variability.

Your results suggest that not normalising prior to training means that the model has to include adaptations to overcome the variation that otherwise would be controlled by normalisation. Because it is a training set it can find very powerful rules to minimise this unwanted variation in the training at so appears to perform well under cross validation.

The fact it falls apart in independent validation shows that whatever rules it had learned to cope with unnormalised data are not generalising to the validation set, I. E. It has an overfitted adaption.

Stability of models in validation sets is far more important than any cross validation performance, so having a standardised normalisation procedure appears to be making a more stable and reliable model for you.

Of course I am assuming errors are rising significantly in the unnormalised data and relatively stable in the normalised

Go with more stable model, with more emphasis on test set performance.

$\endgroup$
2
  • $\begingroup$ You can't base models with "emphasis on test set performance." That type of bias would make it a validation set. $\endgroup$
    – user18764
    Mar 29, 2018 at 13:45
  • $\begingroup$ Yes it would indeed. There should be many stages of developing a model, first training using cross validation to narrow the field to the most likely model, then validating the best one before subjecting the chosen one to independent test sets designed to model real world application. Question describes testing and validation even if calling it a test set. $\endgroup$
    – ReneBt
    Mar 30, 2018 at 2:30
0
$\begingroup$

An example where this can occur is regularised linear regression.

Normalising the data (like converting each regressor in a standard score $X_i^\prime = (X_i-\mu_{X_i})/\sigma_{X_i}$) changes the scale of the parameters. This will have an effect on the coefficients which scale similarly with a same rate. For instance the OLS solution

$$\hat{y} = a + b_1 X_1 + b_2 X_2 + \dots b_n X_n$$ will change into

$$\hat{y} = a^\prime + b_1^\prime X_1^\prime + b_2^\prime X_2^\prime+ \dots + b_n^\prime X_n^\prime$$

with $b_i^\prime = \sigma_{X_i} b_i$.

And if you regularise based on the sizes of the coefficient, which now have changed, then you will get a different training (and testing) result. Effectively you are changing the regularisation term in the cost function

  • A parameter with large $\sigma_{X_i}$ would before the normalisation have a large influence on the model for a small coefficient, and after normalisation this influence will be reduced.

    If the relevant parameters in your model (the ones that are related to the true model or are good general predictors and not noise) are with large $\sigma_{X_i}$ then the normalisation will decrease the quality of the model.

  • In addition the optimum/minimum error in the test data may be achieved with a lower degree of regularisation. (because the bias has become stronger) And this you may observe as the training error improving at a given testing error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.