1
$\begingroup$

Many students come across normalization and standardization as a easy to understand, but hard to grasp/motivate problem. Often times reasons for why we should normalize or standardize data range from the fact we want to do PCA (variance sensitive) or want to make the results more readable. I was wondering if anyone had a simple, nearly trivial example that is teachable to students where the motivation behind normalization or standardization is justified?

$\endgroup$
1
  • $\begingroup$ A client ran into severe numerical problems (total loss of numerical precision in computing sums of squares and products), as well as issues in reporting results, in making simple regressions of monitoring data over time. Her problem stemmed from using her software's default method of representing time as seconds since 1970. That put the values of her times around $10^9$ and the rates of change (which were a few percent per year) were expressed as values less than $10^{-8}$. The solution was not necessarily standardization, but expressing times in years since 2000 worked nicely. $\endgroup$
    – whuber
    Oct 24, 2017 at 16:05

2 Answers 2

1
$\begingroup$

Basically, you want your data to be normalized or standardized so that it would have zero mean and unit-variance. Although another way to explain it would be that the features need to be normalized simply because you do not want some features overpower the other features. Let's say for example that you have a column for movie prices. The prices are around 20-30 dollars each and then you have another column for number of people who went to the cinema that day and the value is around 1000-2000 people. You can clearly see that the column that contains the number of people is clearly larger than the price of the movie hence making it somewhat incomparable. The data is normalized to address this issue and make sure that one feature doesn't overpower the other features so that when it is used in Machine Learning algorithms, it would be comparable and easier to create models from it.

$\endgroup$
1
$\begingroup$

There can be different reasons to normalize/standardize in different contexts, so I don't think it could be distilled down to a single example. Here are a just a few examples:

In linear regression, we might standardize the explanatory variables if they have different units, but we want the weights to have the same units (e.g. to compare them). After standardizing, each weight represents the expected change in the output variable per standard deviation increase of the corresponding explanatory variable (holding all others fixed). We might choose not to standardize if we want the weights to be expressed in terms of the original units. In this context, standardizing simply corresponds to a change of units; it doesn't have any effect on model performance or fundamentally change what we're doing.

Inputs are typically normalized/standardized when using regularization (e.g. l1/l2 penalties, as in lasso/ridge regression). If the inputs weren't normalized, each weight would be penalized differently, depending on the scale of the corresponding variable (we don't want this). In this context, normalization does fundamentally change what we're doing.

For some problems, normalization lets us compute a solution more efficiently. For example, long, narrow valleys in the objective function can arise if inputs to a neural net have very different scales. Gradient-based optimization algorithms can take a long time to converge in this situation, because the negative gradient points mostly across the canyon walls, rather than toward the minimum. Normalizing the inputs can help improve convergence. In this situation, we're typically interested in predictions rather than values of the network parameters themselves, and we use normalization for computational reasons.

Distance-based methods are an interesting case (e.g. k nearest neighbors and RBF kernels). For example, the Euclidean distance between points depends on how we scale each dimension. Dimensions with greater scale contribute more heavily to the distance. It's common practice to normalize the inputs so that each feature contributes equally. In the absence of further information, this seems like a reasonable thing to do. But, there are no guarantees that this is actually the best choice. For example, if the original data contained many small amplitude features that were essentially noise, normalization would amplify the noise and decrease performance. Ideally, we might like highly informative features to have large scale, and less informative ones to have smaller scale (hence, various methods for learning an appropriate scaling).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.