I am currently try to normalize data. But I am not sure the differences between $(x - \mu)/ \sigma$ and $x/\sigma$. What are the advantages and differences of these two approaches?

  • $\begingroup$ The correct, traditional (in statistical analysis) name for the first formula of normalization is "z-standardization". The second formula may have only limited and seldom usage (note that $\sigma$ itself implies centering already, so why not center $x$?). $\endgroup$ – ttnphns Feb 8 '16 at 11:12
  • $\begingroup$ See also this answer keeping "normalize" and "standardize" as separate terms. (However, I don't think it is a common rule; many people use the two words interchangeably.) $\endgroup$ – ttnphns Feb 8 '16 at 11:13

In the first one you center your data both by mean and variance; which means that your normalized data will be a 0-mean collection and have a variance of 1. In the second case, you simply normalize the variance so that you normalized collection will have a unit variance. I'll give you examples of those processes in the case of machine-learning algorithms.

Why is it important and when should I use it?

Normalizing your data with variance (meaning creating a $x'$ that have a variance of 1) is really important most of the time because it scales every dimension of your feature vector to one so that their magnitude become comparable. It means that if you learn an algoithm that gives you weights on your data dimension (say $x=(x_1,x_2)$ and you learn a classifier with weights $w=(w_1,w_2)$) then you can compare those weights and tell that if $w_1 > w_2$ the impact of dimension 1 is higher than the one of dimension 2. I think it should be done most of the times!

Normalizing with mean is slightly subtler. It will change the interpretation of the intercept of a learned model, meaning if you want to predict $y=w_1x_1 + w_2x_2 + b$ then we've seen that if you haven't normalized your data with the variance then you wouldn't be able to compare the weights. If you haven't centered your data with mean you cannot give precise information over $b$ (the intercept). But if you have then you have: $b=\mathbb{E} y$ which is quite a nice result.

Caution with mean-centering: depending on the implementation or framework you use, centering with mean can be harmful because it can break sparsity and get your code to become very inefficient! So take care, sometimes it's better to prototype your algorithms with mean-centering for interpretability but not to do it with your whole dataset when deploying to production for performances.

| cite | improve this answer | |
  • $\begingroup$ +1, especially for the last caution on sparcity break. However, I felt a bit confused by the loose usage of a word, specifically normalizing. What are we "normalizing" in, say $(x - \mu)/ \sigma$? Are we normalizing data $x$ (with st. deviation) or have "normalized" st. deviation $\sigma$ itself? Choose one terminology application, not both. $\endgroup$ – ttnphns Feb 8 '16 at 10:29
  • $\begingroup$ Thanks for the comment, I'll try to make my post clearer indeed with this normalization term. I'd say we've normalized the data $x$ - which results in $x'$ that has a unit variance but the st. deviation $\sigma_x$ hasn't changed though (but $\sigma_{x'}=1$). $\endgroup$ – Vince.Bdn Feb 8 '16 at 10:33
  • 2
    $\begingroup$ Normalize, standardize and scale are all overloaded terms. It's my impression that machine learning people more often use "normalize" for one of these scalings, while statistical people are less likely to use it. Advice from bitter experience: never trust any of these terms to mean anything unless you can see the equation or the code (as you can here). $\endgroup$ – Nick Cox Feb 8 '16 at 10:42
  • $\begingroup$ Agreed. In machine-learning we tend to say normalize all the time but I feel like standardize and scale seldom imply centering by mean though? In statistics I feel like people tend to be more precise and give fancy names (z-standardization) which results in a better understanding and less ambiguity. $\endgroup$ – Vince.Bdn Feb 8 '16 at 10:45
  • $\begingroup$ Vince, yeah. Because of all many "standardizations" in statistics the z-standardization is used a great majority of the time it got its special name "centuries ago", its values are called z-scores (this, however, overlaps with slightly another usage of "z-score" word - a value in specifically standard normal distribution). $\endgroup$ – ttnphns Feb 8 '16 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.