2
$\begingroup$

I have a question on diagonal LDA in the book "Elements of statistical learning" by Hastie, Tibshirani et al. On page 652, it says

The discriminant score of the diagonal-covariance LDA for class $k$ is $$ \delta_{k}(x)= - \sum_{j=1}^{p} \dfrac{(x_{j}-\bar{x}_{kj})^{2}}{s_{j}^{2}} + 2 \log \pi_{k}, $$ [see (4.12) on page 110].

First thing I'm wondering about is the fact that (4.12) is the discriminant function of QDA and not of LDA, I thought diagonal LDA is based on LDA?

Nevertheless, this is (4.12):

$$ \delta_{k}(x) = - \frac{1}{2} \log |\mathbf{\Sigma}_k| - \frac{1}{2}(x-\mu_k)^T\mathbf{\Sigma}_k^{-1}(x-\mu_k)+\log \pi_k $$

$\mathbf{\Sigma}_k$ is the covariance matrix of each class $k$. For the transformation below I used $\mathbf{\Sigma}_k = \mathrm{diag}(\sigma_j^2) \; \forall k$.

So now I would like to justify why dLDA has the above discriminant function. With what I know, I was able to transform (4.12) into

$$\delta_{k}(x) = - \frac{1}{2} \log |\mathrm{diag}(\sigma_j^2)| - \frac{1}{2} \sum\limits_{j=1}^p\frac{(x_j-\mu_k)^2}{s_j^2}+\log \pi_k. $$

Explanation of the variables:

  • $s_j$ is the Standard Deviation of the $j$th gene
  • $\bar{x}_{kj}=\sum_{i \in C_{k}}\frac{x_{ij}}{N_{k}} $ mean of the $N_{k}$ values for gene $j$ in class $k$
  • $C_{k}$ Indexset for class $k$
  • $\pi_k$ Prior probability of class $k$, $\sum_{k=1}^{K}\pi_{k}=1$
  • I think $\mu_k$ = $\bar{x}_{kj}$

So now I would really appreciate some hints how to proceed in order to transform some more so I end up with the discriminant function of dLDA.

$\endgroup$
11
  • $\begingroup$ What does your $\boldsymbol\Sigma = \sigma_j^2 I$ formula mean? Is it $\boldsymbol\Sigma_k$ for all $k$? If so, what does $\sigma_j^2 I$ mean? Diagonal matrix with $\sigma_j^2$ on the $j$-th place on the diagonal? If so, you cannot write it like that. Maybe $\operatorname{diag}(\sigma_j^2)$ or something like that. $\endgroup$
    – amoeba
    Dec 12, 2017 at 15:38
  • $\begingroup$ @amoeba yes, I forgot to state that it's used for all $k$. And also true, it's $diag(\sigma_j^2)$, I will edit it $\endgroup$ Dec 12, 2017 at 17:25
  • $\begingroup$ All right. Can't you compute the determinant of $\operatorname{diag}(\sigma^2_j)$? $\endgroup$
    – amoeba
    Dec 12, 2017 at 19:17
  • $\begingroup$ Err, I haven't thought about this yet. In what way would it help me? I'm mostly confused by the $\frac{1}{2}$ in front of the sum, that's not there anymore later and also I don't know how to handle the $log$ of the Matrix... $\endgroup$ Dec 12, 2017 at 19:28
  • $\begingroup$ It's log of the determinant, that's what vertical bars mean. $\endgroup$
    – amoeba
    Dec 12, 2017 at 19:32

1 Answer 1

3
$\begingroup$

I could solve my problem thanks to amoeba's comments.

My first Transformation got me to:

$\delta_{k}(x) = - \frac{1}{2} \log |diag(\sigma_j^2)| - \frac{1}{2} \sum\limits_{j=1}^{p} \dfrac{(x_{j}-\mu_k)^{2}}{\sigma_{j}^{2}}+\log \pi_k $

Because $\mu_k$ is the vector $(\bar{x}_{k1},...,\bar{x}_{kp})^T$ with $\bar{x}_{kj}$ being the mean of the $j$th gene in class $k$, it actually should have been $\delta_{k}(x) = - \frac{1}{2} \log |diag(\sigma_j^2)| - \frac{1}{2} \sum\limits_{j=1}^{p} \dfrac{(x_{j}-\bar{x}_{kj})^{2}}{\sigma_{j}^{2}}+\log \pi_k $.

The term $- \frac{1}{2} \log |diag(\sigma_j^2)|$ means $- \frac{1}{2} \log det(diag(\sigma_j^2))$ which is a constant. As the covariance matrices of all classes are equal, this constant is the same for all classes. $\delta_k(x)$ is used to compute the discriminant score of each class to then compare them to each other. Hence this constant can be dropped because it changes all scores equally.

For the same reason, the remaining $\delta_{k}(x) = - \frac{1}{2} \sum\limits_{j=1}^{p} \dfrac{(x_{j}-\bar{x}_{kj})^{2}}{\sigma_{j}^{2}}+\log \pi_k $ can be multiplied by $2$ and this equals the discriminant function in the book.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.