In general, does normalization mean to normalize the samples or features?

Question

I'm just getting into machine learning, and I have seen two conflicting practices for normalization. To be concrete, let's suppose that we have a $n \times d$ matrix containing our training data, where $n$ is the number of samples and $d$ is the number of features.

When people say that they normalize their data before running whatever algorithm, I have seen that they do one of the following things:

• normalize the columns of the data matrix so that $A_{1,i}^2 + A_{2, i}^2 + \cdots + A_{n, i}^2 = 1$ for each feature $i$
• normalize the rows of the matrix so that each sample vector has the same norm

In general, when someone refers to normalization of data, which of the following are they referring to?

I was under the impression that it was the first one (seems to make the most sense to me), but looking at the documentation for sklearn's preprocessing library, it appears that the default behavior is the second one. This doesn't make sense to me.

Answer 1

Normalization is much trickier than most people think. Consider categorical and nonlinear predictors. Categorical (multinomial; polytomous) predictors are represented by indicator variables and should not be normalized. For continuous predictors, most relationships are nonlinear, and we fit them by expanding the predictor with nonlinear basis functions. The simplest case is perhaps a quadratic relationship $\beta_{1}x + \beta_{2}x^2$. Do we normalize $x$ by its standard deviation then square the normalized value for the second term? Do we normalize the second term by the standard deviation of $x^2$?

The mere use of normalizing so that the sum of squares for a column equals one, or normalizing by the standard deviation assumes that the predictor is one such that squaring it is the right thing to do. In general this only works correctly when the predictor has a symmetric distribution. For asymmetric distributions, the standard deviation is not an appropriate summary statistic for dispersion. One might just as easily entertain Gini's mean difference or the interquartile range. It's all arbitrary.

Answer 2

In general, normalizing the features of one sample. I would not really talk much about rows and columns here, since the feature matrix can obviously transposed. I almost always span features over the rows as this makes it easier to perform calulations on the matrix in, e.g., C++.

Normalizing along the samples (I think this is your first bullet point) does indeed not make much sense. I think it is sometimes done in Auto-Encoder/Decoder methods (edit: actually only on the weight matrix) when the weights are shared in a particular way.

Think about it like this: if you normalize along the samples, how do you normalize a new sample that should be classified? Using the normalization term you have obtained during training or re-calculating the norm over the training examples + the new examples. Certainly the second one will eventually make the classifier fail. The first one will not guarantee that your normalization sums up to one anymore.

Answer 3

That depends on the analysis steps following the normalization

If nothing else is said, then it commonly refers to normalizing the features under consideration across all samples (e.g. to afterwards classify samples or to predict their value w.r.t to some quantitative attributes, or to conduct dimensionality reduction techniques under the requirement of avoiding some bias introduced by the hetereogeneous range of attributes)

In specific fields however, in particular in analysis of microarray data, normalization along the samples is a widely used preprocessing step to remove unwanted variation during quality control (hopefully mostly technical noise, but it also affects real biological differences of course). You may e.g. want to have a look at https://en.wikipedia.org/wiki/Quantile_normalization.

This normalization technique affects even both directions at the same time (samples and features):

1. Look for the feature with the smallest value within each sample (may be a different attribute for each of the samples)
2. Collect all these smallest values and calculate the average of them
3. Assign this new value to the original places you took it from, so that all samples now have the same value at the attribute that originally showed the smallest value within the respective sample
4. Do the same with the 2nd smallest value, 3rd,... until all data are processed this way

Finally the range of all data is the same for any of the samples. This data set is then the basis for further processing.