The downside is when the noise is much greater for one variable than another. Standardisation matches the variance of each variables. If that balance comes from relevant underlying processes relevant to what you are interested in, great. If the variance comes from noise then you amplify the noise in variables with a lower information content.
Some cases where standardising can be unhelpful :
- Sparse signals, featureless regions will contain just noise, which standardisation will inflate.
- Measurement error is larger for some variables
- Background (unrelated to processes of interest) random variability in some variables is larger
* EDIT *
The original answer assumed that noise variables are on a lower dynamic range than informative variables, which is often the case in signals, where noise is often proportional to the square root of intensity, so low signal areas also have a much lower variance, it is just that a higher proportion of that variance is irrelevant. There is no general requirement that noise has a lower dynamic range than information and so this is dealt with in more detail at the end of this edit.
To address the first comment:
$X$ and $Y$ are usually used for causal or relational models, so $X_1$ and $X_2$ is less confusing in PCA, which does not reference any variables external to the X matrix.
So can we say that the direction of z depends on the correlation among
X and units at the same time and standardization will always tend to
lead the direction toward a 'balance'?
What standardisation does is balance based on variation, not on informativeness. It makes sure a relative value change in $X_1$ accounts for the same fraction of variation as the same relative value change in $X_2$. So yes, it will 'balance' the contribution of $X_1$ and $X_2$ but does so specifically in terms of their variance. How this affects balance of any other property depends on the underlying data generating processes.
For example, if some variables are less informative (have a higher irrelevant variation) and standardising against total variance will imbalance the informativeness. This leads to a less informative model since the disinformation is emphasised. In the ideal case that $X_1$ is noiseless and $X_2$ is pure noise then a unit change in disinformation in $X_2$ now is equivalent to a unit change in information in $X_1$.
Something to bear in mind is that informativeness is very context dependent. What is irrelevant in one context is information about something else, it is just not related to what we are interested in.
And that balance disregards the true relationships between Xs and Y
Correlation is standardised by definition, so standardisation has no impact on correlation between variables. Covariance however is not already standardised and has direct relevance to PC ranking. This means that when we scale our variables we affect the relative contribution of information and disinformation to the total covariance. How this impacts depends on the dynamic range of the information relative to the disinformation.
If we split variance into informative ($V_i$) and uninformative ($V_u$) contributions, then we think of 3 variables:
Noiseless-
$$V_{X_1} = V_i$$
Signal = Noise-
$$V_{X_2} = V_i/2 + V_u/2$$
Noise-
$$V_{X_3} = V_u$$
Then what we are dividing by in each case is $V_i$,$V_i/2 + V_u/2$, and $V_u$. Now if
1. V_u > V_i then standardisation will help, by levelling the contribution of the variance in both variables. Hypothetically the gain will be less than if you had some way of directly scaling against informativeness, but it will be an improvement over the unscaled version.
2. V_u = V_i then standardisation will make no difference, all sums will be the same total variance.
3. V_u < V_i then standardisation will hinder by increasing the contribution of disinformation since you are dividing the noisy variable by a smaller number.
So if we divide by disinformation, then yes it can ignore the underlying relationship between the variables and instead promote the unrelated noise.
