1
$\begingroup$

Let $q\leq p$. As in Tibshirani's statistical learning book, one can describe the PCA problem as optimizing the $q$-dimensional reconstruction error, given on a dataset $\{x_n\}_{n=1}^N$ in $\mathbb{R}^p$ by $$ \inf_{\mu\in \mathbb{R}^p,\lambda \in \mathbb{R}^q,V_qV_q^{\top}=I_d}\, L(\mu,V_q,\lambda) \triangleq \frac1{N}\sum_{n=1}^N\left\| x_n - V_q\lambda - \mu \right\|, $$ for the parameters $\mu\in \mathbb{R}^p$, $\lambda \in \mathbb{R}^q$, and (most notable for this question) the orthogonal matrix $V_q$.

My question is, is there any benefit to further constraining $V_q$ to have determinant $1$ (called a special orthogonal matrix)? That is, do people consider the constrained PCA problem $$ \inf_{\mu\in \mathbb{R}^p,\lambda \in \mathbb{R}^q,V_qV_q^{\top}=I_d,\, \det(V_q)=1}\,L(\mu,V_q,\lambda) \triangleq \frac1{N}\sum_{n=1}^N\left\| x_n - V_q\lambda - \mu \right\|? $$ If so, what is the advantage/interpretation over classical PCA?

$\endgroup$
4
  • $\begingroup$ Isn't it the case that if $q \ne p$, $V$ is not square, so can't have a determinant? $\endgroup$
    – fblundun
    Dec 1 '20 at 14:19
  • 1
    $\begingroup$ Similarly, you say $V_q$ is orthogonal, but non-square matrices can't be orthogonal, only semi-orthogonal. $\endgroup$
    – fblundun
    Dec 1 '20 at 14:35
  • $\begingroup$ Oh right, I meant its columns are orthogonal. So this is thesame as semi-orthogonal no? $\endgroup$
    – BLBA
    Dec 1 '20 at 14:39
  • 2
    $\begingroup$ The difference in formulations is inconsequential, because corresponding to any solution are at least $2^N-1$ other solutions that differ by changing the signs of the principal components; and half of those solutions have the negative of the original determinant. $\endgroup$
    – whuber
    Dec 1 '20 at 15:26
3
$\begingroup$

If $q \ne p$, then $V$ is not a square matrix, so $\det(V)$ is undefined and the proposed additional constraint doesn't work.

If $q = p$, suppose we have a solution to the problem as defined in Tischibirani's book. Then $V$ is orthogonal, so $\det(V) \in \{1, -1\}$. If $\det(V) = 1$ then our solution satisfies the additional constraint already. If $\det(V) = -1$ we can create a solution which satisfies the additional constraint simply by reversing the sign of every element of the first column of $V$ and of the first coordinate of every $\lambda$. So the additional constraint doesn't change much.

$\endgroup$
0
0
$\begingroup$

The wikipedia page: https://en.wikipedia.org/wiki/Orthogonal_matrix has some more information on orthogonal matrices and the meaning of the determinant. It says for example that orthogonal matrices will have determinant -1 or +1 and that a +1 determinant implies that the matrix acts as a rotation. Meanwhile, with either determinant included the orthogonal matrix may represent a "rotation, reflection or rotoreflection".

Others have noted however that for every solution to the PCA problem (which will have det -1 or 1) there is a simple sign flipping way to find a solution with the det of the opposite sign. The implication is that whatever the solution is in the less constrained problem it must have an equivalent in the constrained one, so perhaps it is the case that the constrained problem usefully reduces the solution space whilst keeping it optimal.

$\endgroup$
4
  • $\begingroup$ Interesting. But, this makes me wonder, if the goal is to find the optimal hyperplane why not just leave $V_q$ an unconstrained $q\times p$-dimensional matrix? $\endgroup$
    – BLBA
    Dec 1 '20 at 14:22
  • 2
    $\begingroup$ This answer is misleading, because all orthogonal matrices automatically preserve volume anyway. $\endgroup$
    – whuber
    Dec 1 '20 at 15:27
  • $\begingroup$ Thanks for the comment @whuber. I wasn't aware of the fact. I've modified the answer to integrate the new information. $\endgroup$
    – emiru
    Dec 1 '20 at 22:46
  • $\begingroup$ That helps, thank you. But please reassess your conclusion about suboptimality in light of the accepted answer by fblundun. $\endgroup$
    – whuber
    Dec 1 '20 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.