0
$\begingroup$

When and why do we need standardization for machine learning? What's the benefit for doing so?

For example, we replace $y_i$ with $(y_i - u_y)/ \sigma _y $ we also replace $x_i$ with $x_ij - u_j / \sigma_j$

It would be great if someone provides me some intuitions.

$\endgroup$
  • 1
    $\begingroup$ Asking about mean=0, sd=1 across all machine learning is a bit broad. Do you have a specific question about this? $\endgroup$ – Sycorax says Reinstate Monica Jun 10 '17 at 21:38
2
$\begingroup$

The short answer is that you generally need to do some kind of gradient descent to train your model, and this will rely on selecting initial conditions. Poor initial conditions will lead to poor convergence results. As an example, consider a single neuron in a neural network $\sigma(ax+b)$ with only one feature, where say $\sigma$ is a ReLu. In this case, if you intialize $a,b$ to be $N(0,1)$, they will have magnitudes on the order of 1. Now imagine your inputs are data with $x$ being around $-100$. Most of your activations will be zero and hence gradient descent becomes impossible. You could remedy this with a leaky ReLu, but then the training would be slow. Whereas if you normalize your data to be commensurate with your initial conditions, you'll have nonzero gradients.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.