Univariate vs. Multivariate Standardization

Question

There are several common methods for scaling input features to machine learning models prior to training the model. The most popular methods seem to be standardization (centering by the mean and dividing by the standard deviation of the variable) and min-max normalization (centering by the minimum value and dividing by the range of the variable).

While reviewing information on this topic I've only come across univariate methods and it begs the question of why multivariate aren't very prevalent? For example rather than performing standardization to each variable in a univariate way, it could be performed in a multivariate way. $$ \boldsymbol{z}=\boldsymbol{\Sigma}^{-1/2}(\boldsymbol{y}-\boldsymbol{\mu}) $$

Where $\boldsymbol{y}$ is a vector of original unscaled input features (single observation of training data), $\boldsymbol{\mu}$ is a vector containing the main of the unscaled training data input features, $\boldsymbol{\Sigma}$ is the correlation matrix of training data input features and $\boldsymbol{z}$ is the transformed vector of input features.

This should lead to the following for the training data:

1. Each transformed feature having a mean of 0.
2. Each transformed feature having a standard deviation of 1.
3. Each each transformed feature being uncorrelated to other features.

Is there a reason this kind of preprocessing isn't more prevalent?

EDIT: For this question I'm particularly interested in model accuracy and not inference.

Answer 1

the way you define these two, you can think of univariate standardization as Euclidian and multivariate as Mahalanobisian.

The arguments against the latter would be that you have to estimate the covariance matrix, and for a large number of dimensions it is between problematic and impossible. The number of elements in the covariance matrix is $O(n^2)$ where n is number of features. Hence, with a large number of dimensions you quickly flood the matrix with distortions. Plus, this standardization assumes linear relationship between features. So, even if you have a nonlinear model your data gets linearized in some way. Finally, it's very sensitive to outliers. An outlier in one dimension can mess up entire matrix, while in univariate case it only spoils one feature.

Answer 2

Consider why someone would standardize at all. Two reasons come to mind.

1. Computational

When people standardize to ease the computations, it is typically because numerical optimization gets confused when features are on dramatically different scales. The multivariate standardization does not help with this, as univariate standardization already puts variables on the same scale (in some sense).

2. Interpretation

Standardization can help with interpretation by giving the analyst the ability to easily say what happens to the outcome when an input changes by some number of standard deviations. Multivariate standardization does not help with this.

Next, highly correlated features can be problematic when it comes to doing the math on a computer (more computational issues), but that typically requires a huge amount of correlation. Further, there are ways for there to be considerable dependence with minimal or zero correlation. Your multivariate standardization does not address such dependence.

Finally, removing the correlation between features means that your model features are all blends of the original features. Correlated features can make it harder to draw causal inferences, sure, but it’s also hard to interpret models when the features are linear combinations of the original measurements.

Multivariate standardization is completely reasonable. Indeed, what you’ve proposed has a strong relationship with principal components analysis. However, I hope this gives a sense of why people often do not care to make it so complicated and just stick to univariate standardization.