If a multi-class Linear Discriminant Analysis (or I also read Multiple Discriminant Analysis sometimes) is used for dimensionality reduction (or transformation after dimensionality reduction via PCA), I understand that in general a "Z-score normalization" (or standardization) of features won't be necessary, even if they are measured on completely different scales, correct? Since LDA contains a term similar to the Mahalanobis distance which is already implying normalized Euclidean distances?

So it would not only be not necessary, but the results after an LDA on standardized and non-standardized features should be exactly the same!?

  • 1
    $\begingroup$ in general a "Z-score normalization" (or standardization) of features won't be necessary, even if they are measured on completely different scales No, this statement is incorrect. The issue of standardization with LDA is the same as in any multivariate method. For example, PCA. Mahalanobis distance has nothing to do with that topic. $\endgroup$ – ttnphns Jul 23 '14 at 18:46
  • $\begingroup$ Thanks, it would be great if you could maybe comment on this "standardization issue" in PCA, for example. If features are not standardized for PCA, aren't some features contributing (weighted) more if they are measured on a different scale and giving me completely different component axes? And for the LDA, why won't it be necessary? Is the outcome (the linear discriminants) different, if not, why? $\endgroup$ – user39663 Jul 23 '14 at 18:59
  • 2
    $\begingroup$ When you standardize (i.e. center, then scale) you will actually be analyzing correlations. If you don't standardize, only center, you will actually be analyzing covariances. Results will differ, which is normal, because it is like you deal with different data. This fact shouldn't worry you. You may enjoy reading thread stats.stackexchange.com/q/62677/3277. $\endgroup$ – ttnphns Jul 23 '14 at 19:16
  • 2
    $\begingroup$ @SebastianRaschka, amoeba: I must reconsider my comment The issue of standardization with LDA is the same as in any multivariate method. Actually, with LDA (as opposed to PCA, for example) results should not differ whether you've only centered (LDA internally always centers variables, to extract discriminants) or z-standardized the data. $\endgroup$ – ttnphns Jul 29 '14 at 10:08
  • 2
    $\begingroup$ (Cont.) Eigenvalues, standardized coefficients, structure correlations, discriminant scores - everything will be the same. Only eigenvectors will differ. The reason why there's no effect of standardization on the main results in LDA is that LDA decomposes ratio of Between-to-Within covariances, and not the covariance itself having its magnitude (as PCA does). $\endgroup$ – ttnphns Jul 29 '14 at 10:12

The credit for this answer goes to @ttnphns who explained everything in the comments above. Still, I would like to provide an extended answer.

To your question: Are the LDA results on standardized and non-standardized features going to be exactly the same? --- the answer is Yes. I will first give an informal argument, and then proceed with some math.

Imagine a 2D dataset shown as a scatter plot on one side of a balloon (original balloon picture taken from here): LDA on a baloon

Here red dots are one class, green dots are another class, and black line is LDA class boundary. Now rescaling of $x$ or $y$ axes corresponds to stretching the balloon horizontally or vertically. It is intuitively clear that even though the slope of the black line will change after such stretching, the classes will be exactly as separable as before, and the relative position of the black line will not change. Each test observation will be assigned to the same class as before the stretching. So one can say that stretching does not influence the results of LDA.

Now, mathematically, LDA finds a set of discriminant axes by computing eigenvectors of $\mathbf{W}^{-1} \mathbf{B}$, where $\mathbf{W}$ and $\mathbf{B}$ are within- and between-class scatter matrices. Equivalently, these are generalized eigenvectors of the generalized eigenvalue problem $\mathbf{B}\mathbf{v}=\lambda\mathbf{W}\mathbf{v}$.

Consider a centred data matrix $\mathbf{X}$ with variables in columns and data points in rows, so that the total scatter matrix is given by $\mathbf{T}=\mathbf{X}^\top\mathbf{X}$. Standardizing the data amounts to scaling each column of $\mathbf{X}$ by a certain number, i.e. replacing it with $\mathbf{X}_\mathrm{new}= \mathbf{X}\boldsymbol\Lambda$, where $\boldsymbol\Lambda$ is a diagonal matrix with scaling coefficients (inverses of the standard deviations of each column) on the diagonal. After such a rescaling, the scatter matrix will change as follows: $\mathbf{T}_\mathrm{new} = \boldsymbol\Lambda\mathbf{T}\boldsymbol\Lambda$, and the same transformation will happen with $\mathbf{W}_\mathrm{new}$ and $\mathbf{B}_\mathrm{new}$.

Let $\mathbf{v}$ be an eigenvector of the original problem, i.e. $$\mathbf{B}\mathbf{v}=\lambda\mathbf{W}\mathbf{v}.$$ If we multiply this equation with $\boldsymbol\Lambda$ on the left, and insert $\boldsymbol\Lambda\boldsymbol\Lambda^{-1}$ on both sides before $\mathbf{v}$, we obtain $$\boldsymbol\Lambda\mathbf{B}\boldsymbol\Lambda\boldsymbol\Lambda^{-1}\mathbf{v}=\lambda\boldsymbol\Lambda\mathbf{W}\boldsymbol\Lambda\boldsymbol\Lambda^{-1}\mathbf{v},$$ i.e. $$\mathbf{B}_\mathrm{new}\boldsymbol\Lambda^{-1}\mathbf{v}=\lambda\mathbf{W}_\mathrm{new}\boldsymbol\Lambda^{-1}\mathbf{v},$$ which means that $\boldsymbol\Lambda^{-1}\mathbf{v}$ is an eigenvector after rescaling with exactly the same eigenvalue $\lambda$ as before.

So discriminant axis (given by the eigenvector) will change, but its eigenvalue, that shows how much the classes are separated, will stay exactly the same. Moreover, projection on this axis, that was originally given by $\mathbf{X}\mathbf{v}$, will now be given by $ \mathbf{X}\boldsymbol\Lambda (\boldsymbol\Lambda^{-1}\mathbf{v})= \mathbf{X}\mathbf{v}$, i.e. will also stay exactly the same (maybe up to a scaling factor).

  • 2
    $\begingroup$ +1. The "moral" of the whole story is that the difference between the only centered data $\bf X$ and the standardized data $\bf X \Lambda$ is absurbed entirely in the eigenvectors. So when the data are multiplied by the corresponding eigenvectors to produce discriminant scores, the effect $\bf \Lambda$ of the standardization cancels out. $\endgroup$ – ttnphns Aug 6 '14 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy