3
$\begingroup$

I understand that feature scaling speeds up gradient descent by making the geometry of the cost function more hospitable (i.e. more circular), but why is mean normalization useful? Doesn't it just "move" the cost function closer to the origin?

$\endgroup$
1
$\begingroup$

Normalization of data is a tricky process. You have your terminology a little bit mixed up, but your intuition seems correct. What you refer to as normalization is in fact the centering of data. This is beneficial for certain algorithms (PCA for instance), or as @Alex R. pointed out for activation functions for neural networks. It does not have any bearing on gradient descent itself.

The process of normalization is in fact what you refer to as scaling. As you correctly point out, normalizing the data matrix will ensure that the problem is well conditioned so that you do not have a massive variance in the scale of your dimensions. This makes optimization using first order methods (i.e. gradient descent) feasible.

Your confusion may stem from the fact that the process of normalizing and centering your data is sometimes referred to as standardizing the data.

$\endgroup$
  • $\begingroup$ Ok, that makes sense, and clears up my confusion. (I now realize I was confused because in the video by Andrew Ng I was watching on feature scaling and its usefulness in gradient descent, he sort of tacks on centering of data (or what he calls "mean normalization") without giving it context. Video is at youtube.com/watch?v=cOrx5-9-YjQ ) $\endgroup$ – Tyler Nov 23 '17 at 4:46
1
$\begingroup$

Your wording is bizarre. Normalization makes your input commensurate in size with the initializations you pick for your weights/parameters. The mean shift balances your input to have both negatives and positives, which is important when your model has asymmetric functions, such as ReLU neurons or the sigmoid of a logistic regression. It also prevents your functions from being saturated, I.e. have vanishing gradient

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.