2
$\begingroup$

I know the downside of not doing standardization beforehand when the variables are not measured in the same unit. However, what I am wondering is the downside of doing the standardization:

Suppose we have two characteristics measured in the same unit, we assign weights to variables according to their magnitude of variation, in an effort to maximize the sum of squared projection variation.

However, if the characteristics are measured in different units and we choose to standardize (z score) them into the same unit, the PCA will assign the same weight to the two variables at least in 2-dimension and thus we can't differentiate their explanatory power.

Also, should we assign the weight calculated based on the standardized variables to the standardized variables or to the original variables?

Thanks in advance!

$\endgroup$
2
  • $\begingroup$ When you say to standardize the variables, do you mean z-score? Also, consider what kind of correlation these variables have with each other and especially with the other variables. $\endgroup$
    – Dave
    Commented Feb 8, 2020 at 23:19
  • $\begingroup$ Since the correlation is already the standardized metrics, it won't be affected by whether we standardize the variables right? I think the scale and correlation determine the direction at the same time.. $\endgroup$
    – Yuan
    Commented Feb 9, 2020 at 0:17

1 Answer 1

1
$\begingroup$

The downside is when the noise is much greater for one variable than another. Standardisation matches the variance of each variables. If that balance comes from relevant underlying processes relevant to what you are interested in, great. If the variance comes from noise then you amplify the noise in variables with a lower information content.

Some cases where standardising can be unhelpful :

  1. Sparse signals, featureless regions will contain just noise, which standardisation will inflate.
  2. Measurement error is larger for some variables
  3. Background (unrelated to processes of interest) random variability in some variables is larger

* EDIT *

The original answer assumed that noise variables are on a lower dynamic range than informative variables, which is often the case in signals, where noise is often proportional to the square root of intensity, so low signal areas also have a much lower variance, it is just that a higher proportion of that variance is irrelevant. There is no general requirement that noise has a lower dynamic range than information and so this is dealt with in more detail at the end of this edit.

To address the first comment:

$X$ and $Y$ are usually used for causal or relational models, so $X_1$ and $X_2$ is less confusing in PCA, which does not reference any variables external to the X matrix.

So can we say that the direction of z depends on the correlation among X and units at the same time and standardization will always tend to lead the direction toward a 'balance'?

What standardisation does is balance based on variation, not on informativeness. It makes sure a relative value change in $X_1$ accounts for the same fraction of variation as the same relative value change in $X_2$. So yes, it will 'balance' the contribution of $X_1$ and $X_2$ but does so specifically in terms of their variance. How this affects balance of any other property depends on the underlying data generating processes.

For example, if some variables are less informative (have a higher irrelevant variation) and standardising against total variance will imbalance the informativeness. This leads to a less informative model since the disinformation is emphasised. In the ideal case that $X_1$ is noiseless and $X_2$ is pure noise then a unit change in disinformation in $X_2$ now is equivalent to a unit change in information in $X_1$.

Something to bear in mind is that informativeness is very context dependent. What is irrelevant in one context is information about something else, it is just not related to what we are interested in.

And that balance disregards the true relationships between Xs and Y

Correlation is standardised by definition, so standardisation has no impact on correlation between variables. Covariance however is not already standardised and has direct relevance to PC ranking. This means that when we scale our variables we affect the relative contribution of information and disinformation to the total covariance. How this impacts depends on the dynamic range of the information relative to the disinformation.

If we split variance into informative ($V_i$) and uninformative ($V_u$) contributions, then we think of 3 variables:

Noiseless- $$V_{X_1} = V_i$$ Signal = Noise- $$V_{X_2} = V_i/2 + V_u/2$$ Noise- $$V_{X_3} = V_u$$

Then what we are dividing by in each case is $V_i$,$V_i/2 + V_u/2$, and $V_u$. Now if 1. V_u > V_i then standardisation will help, by levelling the contribution of the variance in both variables. Hypothetically the gain will be less than if you had some way of directly scaling against informativeness, but it will be an improvement over the unscaled version. 2. V_u = V_i then standardisation will make no difference, all sums will be the same total variance. 3. V_u < V_i then standardisation will hinder by increasing the contribution of disinformation since you are dividing the noisy variable by a smaller number.

So if we divide by disinformation, then yes it can ignore the underlying relationship between the variables and instead promote the unrelated noise.

$\endgroup$
3
  • $\begingroup$ Thanks! So can we say that the direction of z depends on the correlation among X and units at the same time and standardization will always tend to lead the direction toward a 'balance'? And that balance disregards the true relationships between Xs and Y $\endgroup$
    – Yuan
    Commented Feb 10, 2020 at 8:20
  • $\begingroup$ Thank you so much @ReneBt ! So the root of the uncertainty of whether to do standardization on PCA is that we don't know the correlation of X and Y right? Therefore in this sense, if doing partial least square is more helpful? $\endgroup$
    – Yuan
    Commented Feb 10, 2020 at 21:30
  • $\begingroup$ We can calculate the correlation easily, it is a standardised measure. Z score standardised data is the correlation matrix multiplied by the number of variables. The problem is we can't tell from the X data what variation is informative about what we are externally interested in. If you have external information in the form of Y then PLS will indeed use this to balance the description of X variance and the description of information associated with Y. This is precisely why PLS was developed. $\endgroup$
    – ReneBt
    Commented Feb 11, 2020 at 6:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.