8
$\begingroup$

I have some basic questions regarding PCA (principal component analysis) and LDA (linear discriminant analysis):

  1. In PCA there is a way to calculate the proportion of variance explained. Is it also possible for LDA? If so, how?

  2. Is the “Proportion of trace” output from the lda function (in R MASS library) equivalent to the “proportion of variance explained”?

$\endgroup$
  • 1
    $\begingroup$ Your first question may be a duplicate of stats.stackexchange.com/questions/22569, where you can find answers. Presumably "LDA" means Linear Discriminant Analysis (it has other statistical meanings too, which is why we try to expand acronyms). $\endgroup$ – whuber Aug 13 '13 at 21:45
  • 1
    $\begingroup$ In a sense, a discriminant accounts for a variability as a p. component does, the eigenvalue being the amount of it. However, the "variability" in LDA is of special sort - it is the ratio of between-class variabilty to the within-class variability. Each discriminant tries to account for as much as possible of that ratio. Read further $\endgroup$ – ttnphns Aug 14 '13 at 8:17
  • $\begingroup$ Thanks for the explanation. Therefore, if in the axes of the PC components I label them as “PC (X% of explained Variance)” what would be the correct short term when I label the LDs. Thanks again. $\endgroup$ – wrek Aug 14 '13 at 8:56
  • $\begingroup$ With LDA, the correct wording will be “LD (X% of explained between-group Variance)”. $\endgroup$ – ttnphns Aug 14 '13 at 11:11
  • $\begingroup$ Thanks again for the great help and patience. BTW how can I access the Proportion of trace (LD1, LD2) as I wish to save them in two separate variables? $\endgroup$ – wrek Aug 14 '13 at 12:16
10
$\begingroup$

I will first provide a verbal explanation, and then a more technical one. My answer consists of four observations:

  1. As @ttnphns explained in the comments above, in PCA each principal component has certain variance, that all together add up to 100% of the total variance. For each principal component, a ratio of its variance to the total variance is called the "proportion of explained variance". This is very well known.

  2. On the other hand, in LDA each "discriminant component" has certain "discriminability" (I made these terms up!) associated with it, and they all together add up to 100% of the "total discriminability". So for each "discriminant component" one can define "proportion of discriminability explained". I guess that "proportion of trace" that you are referring to, is exactly that (see below). This is less well known, but still commonplace.

  3. Still, one can look at the variance of each discriminant component, and compute "proportion of variance" of each of them. Turns out, they will add up to something that is less than 100%. I do not think that I have ever seen this discussed anywhere, which is the main reason I want to provide this lengthy answer.

  4. One can also go one step further and compute the amount of variance that each LDA component "explains"; this is going to be more than just its own variance.


Let $\mathbf{T}$ be total scatter matrix of the data (i.e. covariance matrix but without normalizing by the number of data points), $\mathbf{W}$ be the within-class scatter matrix, and $\mathbf{B}$ be between-class scatter matrix. See here for definitions. Conveniently, $\mathbf{T}=\mathbf{W}+\mathbf{B}$.

PCA performs eigen-decomposition of $\mathbf{T}$, takes its unit eigenvectors as principal axes, and projections of the data on the eigenvectors as principal components. Variance of each principal component is given by the corresponding eigenvalue. All eigenvalues of $\mathbf{T}$ (which is symmetric and positive-definite) are positive and add up to the $\mathrm{tr}(\mathbf{T})$, which is known as total variance.

LDA performs eigen-decomposition of $\mathbf{W}^{-1} \mathbf{B}$, takes its non-orthogonal (!) unit eigenvectors as discriminant axes, and projections on the eigenvectors as discriminant components (a made-up term). For each discriminant component, we can compute a ratio of between-class variance $B$ and within-class variance $W$, i.e. signal-to-noise ratio $B/W$. It turns out that it will be given by the corresponding eigenvalue of $\mathbf{W}^{-1} \mathbf{B}$ (Lemma 1, see below). All eigenvalues of $\mathbf{W}^{-1} \mathbf{B}$ are positive (Lemma 2) so sum up to a positive number $\mathrm{tr}(\mathbf{W}^{-1} \mathbf{B})$ which one can call total signal-to-noise ratio. Each discriminant component has a certain proportion of it, and that is, I believe, what "proportion of trace" refers to. See this answer by @ttnphns for a similar discussion.

Interestingly, variances of all discriminant components will add up to something smaller than the total variance (even if the number $K$ of classes in the data set is larger than the number $N$ of dimensions; as there are only $K-1$ discriminant axes, they will not even form a basis in case $K-1<N$). This is a non-trivial observation (Lemma 4) that follows from the fact that all discriminant components have zero correlation (Lemma 3). Which means that we can compute the usual proportion of variance for each discriminant component, but their sum will be less than 100%.

However, I am reluctant to refer to these component variances as "explained variances" (let's call them "captured variances" instead). For each LDA component, one can compute the amount of variance it can explain in the data by regressing the data onto this component; this value will in general be larger than this component's own "captured" variance. If there is enough components, then together their explained variance must be 100%. See my answer here for how to compute such explained variance in a general case: Principal component analysis "backwards": how much variance of the data is explained by a given linear combination of the variables?

Here is an illustration using the Iris data set (only sepal measurements!): PCA and LDA of the sepal measurements of the Iris data set Thin solid lines show PCA axes (they are orthogonal), thick dashed lines show LDA axes (non-orthogonal). Proportions of variance explained by the PCA axes: $79\%$ and $21\%$. Proportions of signal-to-noise ratio of the LDA axes: $96\%$ and $4\%$. Proportions of variance captured by the LDA axes: $48\%$ and $26\%$ (i.e. only $74\%$ together). Proportions of variance explained by the LDA axes: $65\%$ and $35\%$.

\begin{array}{lcccc} & \text{LDA axis 1} & \text{LDA axis 2} & \text{PCA axis 1} & \text{PCA axis 2} \\ \text{Captured variance} & 48\% & 26\% & 79\% & 21\% \\ \text{Explained variance} & 65\% & 35\% & 79\% & 21\% \\ \text{Signal-to-noise ratio} & 96\% & 4\% & - & - \\ \end{array}


Lemma 1. Eigenvectors $\mathbf{v}$ of $\mathbf{W}^{-1} \mathbf{B}$ (or, equivalently, generalized eigenvectors of the generalized eigenvalue problem $\mathbf{B}\mathbf{v}=\lambda\mathbf{W}\mathbf{v}$) are stationary points of the Rayleigh quotient $$\frac{\mathbf{v}^\top\mathbf{B}\mathbf{v}}{\mathbf{v}^\top\mathbf{W}\mathbf{v}} = \frac{B}{W}$$ (differentiate the latter to see it), with the corresponding values of Rayleigh quotient providing the eigenvalues $\lambda$, QED.

Lemma 2. Eigenvalues of $\mathbf{W}^{-1} \mathbf{B} = \mathbf{W}^{-1/2} \mathbf{W}^{-1/2} \mathbf{B}$ are the same as eigenvalues of $\mathbf{W}^{-1/2} \mathbf{B} \mathbf{W}^{-1/2}$ (indeed, these two matrices are similar). The latter is symmetric positive-definite, so all its eigenvalues are positive.

Lemma 3. Note that covariance/correlation between discriminant components is zero. Indeed, different eigenvectors $\mathbf{v}_1$ and $\mathbf{v}_2$ of the generalized eigenvalue problem $\mathbf{B}\mathbf{v}=\lambda\mathbf{W}\mathbf{v}$ are both $\mathbf{B}$- and $\mathbf{W}$-orthogonal (see e.g. here), and so are $\mathbf{T}$-orthogonal as well (because $\mathbf{T}=\mathbf{W}+\mathbf{B}$), which means that they have covariance zero: $\mathbf{v}_1^\top \mathbf{T} \mathbf{v}_2=0$.

Lemma 4. Discriminant axes form a non-orthogonal basis $\mathbf{V}$, in which the covariance matrix $\mathbf{V}^\top\mathbf{T}\mathbf{V}$ is diagonal. In this case one can prove that $$\mathrm{tr}(\mathbf{V}^\top\mathbf{T}\mathbf{V})<\mathrm{tr}(\mathbf{T}),$$ QED.

$\endgroup$
  • 1
    $\begingroup$ +1. Many things you discuss here were covered, slightly more compressed, in my answer. I have added a link to your present answer in the body of that my old one. $\endgroup$ – ttnphns Aug 6 '14 at 10:30
  • 1
    $\begingroup$ @ttnphns: I remember that answer of yours (it has my +1 from long time ago), but did not look there when writing this answer, so many things are indeed presented very similarly, perhaps too much. The main reason I wrote this answer, however, was to discuss "explained variance" (in the PCA sense) of the LDA components. I am not sure how useful it is in practice, but I was often wondering about it before, and have recently struggled for some time to prove the inequality from Lemma 4 that in the end was proved for me on Math.SE. $\endgroup$ – amoeba Aug 6 '14 at 10:47
  • $\begingroup$ Note that the diagonal of $\mathbf{V}^\top\mathbf{T}\mathbf{V}$ is $\lambda+1$, the denominator to compute canonical correlations. $\endgroup$ – ttnphns Aug 6 '14 at 11:54
  • $\begingroup$ @ttnphns: Hmmm... I think that for each eigenvector $\mathbf{v}$, $$B/W = \frac{\mathbf{v}^\top\mathbf{B}\mathbf{v}}{\mathbf{v}^\top\mathbf{W}\mathbf{v}} = \lambda$$ and $$B/T = \frac{\mathbf{v}^\top\mathbf{B}\mathbf{v}}{\mathbf{v}^\top\mathbf{T}\mathbf{v}} = \frac{\mathbf{v}^\top\mathbf{B}\mathbf{v}}{(\mathbf{v}^\top\mathbf{B}\mathbf{v}+\mathbf{v}^\top\mathbf{W}\mathbf{v})} = \frac{\lambda}{\lambda+1},$$ as you say in your linked answer. But the value of $\mathbf{v}^\top\mathbf{T}\mathbf{v}$ (outside of any ratio) cannot really be expressed with $\lambda$ only. $\endgroup$ – amoeba Aug 6 '14 at 12:06
  • $\begingroup$ It appears to me that the eigenvector of a given discriminant contains information of $B/W$ for that discriminant; when we calibrate it with $\bf T$ which keeps the covariances between the variables, we can arrive at the eigenvalue of the discriminant. Thus, the information on $B/W$'s is stored in eigenvectors, and it is "standardized" to the form corresponding to no correlations between the variables. $\endgroup$ – ttnphns Aug 6 '14 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.