2
$\begingroup$

Say we have a classification problem with $k$ classes, an $N\times p$ matrix $X$ whose rows are predictors and an $N\times k$ indicator response matrix $Y$. The problem is to show that LDA on $X(X^TX)^{-1}X^TY$ is equivalent to LDA on $X$.

Algebraically this is trivial if $(X^TX)^{-1}X^TY$ is square and invertible, but of course this need not be the case. I have a rough intuitive understanding of why this should work but the details elude me.

Let $B=(X^TX)^{-1}X^TY$ and let $\Sigma$ be the covariance matrix of the input space of $p$-vectors. LDA on an input $x$ is determined by the quantity $$\log\frac{P(G=k|X=x)}{P(G=\ell|X=x)}=\log\frac{\pi_k}{\pi_\ell}-\frac12(\mu_k+\mu_\ell)\Sigma^{-1}(\mu_k-\mu_\ell)+x^T\Sigma^{-1}(\mu_k-\mu_\ell)$$ (If my notation requires explanation let me know.) Using the transformation specified in the problem changes $\Sigma^{-1}$ to $B(B^T\Sigma B)^{-1}B^T$, which is simply $\Sigma^{-1}$ when $B$ is square and invertible. In general it won't be though. But will the resulting scalar quantity be the same anyway?

Improve this question
$\endgroup$
4
  • $\begingroup$ Could you clarify what you mean by "LDA on $X$," which makes no reference to any responses at all, and by the similar phrase "LDA on $X(X^TX)^{-1}X^TY$," which--since it is the projection of the responses into a subspace--makes no reference to any predictors beyond the space they span? $\endgroup$
    – whuber ♦
    Commented Feb 26, 2017 at 20:18
  • 1
    $\begingroup$ @whuber In both cases the response matrix is $Y$. The $i$th row is classified according to the position of the $1$ in the $i$th row of $Y$. $\endgroup$
    – Matt Samuel
    Commented Feb 26, 2017 at 20:22
  • $\begingroup$ @whuber and the rows of $X(X^TX)^{-1}X^{-1}Y$ are in natural correspondence with the rows of $X$, the predictors, given that they are the output of a linear regression of $Y$ on $X$, so they approximate the indicator response for the corresponding row of $X$. $\endgroup$
    – Matt Samuel
    Commented Feb 26, 2017 at 20:29
  • $\begingroup$ I completed this problem, and my solution was awful (so much that I'm sure I'm missing something myself). One thing is that in fact, $\left(X^T X\right)^{-1}X^TY$ and your new $\Sigma^{-1}$ (with the $B$ terms) will never be invertible, because $X^TY$ is always a loss in rank if your data is centered and if $Y$ is an indicator response matrix (try and see why!). So you'll need something else. $\endgroup$
    – Kevin
    Commented Mar 27, 2018 at 5:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.