For deep neural networks or any ML algorithms , lets say the mean of my dataset is 5 and instead I normalize with 6 or 4, something slightly off. What affect does this have on learning?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ You mean shifting mean of target variable? $\endgroup$– Tim ♦Feb 12, 2019 at 17:50
-
$\begingroup$ I mean normalizing with a slightly incorrect mean $\endgroup$– Rahul DeoraFeb 12, 2019 at 18:18
-
$\begingroup$ What do you mean by "standarizing"? You mean subtracting the mean and adding new mean $(x - \bar x) + \mu_\text{new}$? $\endgroup$– Tim ♦Feb 12, 2019 at 18:31
-
$\begingroup$ I think you should restrict your question to a smaller class of learning problems and algorithms $\endgroup$– DeltaIVFeb 13, 2019 at 7:39
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Nothing remarkable would happen. I assume that by by "standardizing" you mean mean linear transformation that transforms random variable $Y$ to have mean of zero and unit standard deviation by subtracting mean $\bar Y$ and dividing by standard deviation $s$:
$$ Z = \frac{Y-\bar Y}{s} $$
"Standarizing with shifted mean" is presumably adding the new mean $\mu$ i.e. $Z' = Z + \mu$. How does this affect machine learning algorithms?
In case of linear regression, if your model was
$$ Z = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots +\beta_k X_k + \varepsilon $$
than it becomes
$$ Z' = Z + \mu = \tilde{\beta_0} + \beta_1 X_1 + \beta_2 X_2 + \dots +\beta_k X_k + \varepsilon $$
where the intercept is simply shifted by the new mean $\tilde{\beta_0} = \beta_0 + \mu$ (you add $\mu$ on both sides of equation). As about changing the standard deviation of $Z$, this would lead to change of the standard deviation of residuals and would not affect the predictions.
In neural networks, this should lead to change in bias terms of the output layer for the same reason as above. In case of regression trees and random forest, they calculate conditional means, so to correct the results you would only need to shift them by $\mu$, since what the classification and regression trees do, is they make splits based on individual features and calculate means within the groups created using on the splits. So if all values of $Z$ where shifted by constant, then the means within the groups would also shift by the same constant. With $k$ nearest neighbors regression you would calculate mean of $k$ closest points from the training set, so if they all were shifted by $\mu$, their mean would be shifted by $\mu$. No matter of your algorithm, if you wanted to adapt algorithm trained to predict $Z$ to scenario where $Z$ is shifted by $\mu$ (it is the simplest case of distribution shift between training and test data sets), then simply shift the predictions by the same constant $\hat Z+\mu$.
In this thread you can see other example of what happens with relations between variables when you normalize them (TL;DR nothing). Basically, standardization of data is the kind of transformation that in theory should not be needed and should rather not affect the machine learning algorithms, but in some cases it simply makes things computationally easier and because of this is recommended. But the take away message is that is shouldn't matter and it is not something to bother.
Disclaimer: many of the algorithms depend on some kind of randomization during training (e.g. random initialization of weights, random subsampling in random forest etc.), so if you trained the algorithm first on the original data and then on shifted data, you could see also same other differences, but they should be negligible.
