# Principal component analysis, the relation between the variance and error

As is known, the rank-1 PCA aims to solve the following optimization problem $$\min_{x\in\mathbb{R}^d}\quad -x^T \Sigma x\quad\quad\quad s.t.\quad \Vert x\Vert_{2}=1,$$ where $$\Sigma\in\mathbb{S}^{d}$$ is the covariance matrix. Thus the optimum $$x^*$$ of the PCA problem is the top unit eigenvector of $$\Sigma$$. Given an approximation $$\tilde{x}$$ (normalized), the error between the $$\tilde{x}$$ and $$x^*$$ is measured by the sine function $$\sin^{2}(\tilde{x}, x^*) = 1-(\tilde{x}^T x^*)^2.$$ I was wondering if there exists any analytical relationship between the objective function $$\tilde{x}^\top\Sigma \tilde{x}$$ and the error $$\sin^2(\tilde{x}, x^*)$$? Any help appreciated.

## 1 Answer

To long to be a comment:

I don't know what kind of connection are you looking exactly but you may find the following useful. First, why do you measure the error using that function? Note that if you compute the absolute error $$\|\tilde{x}-x^*\|^2 = (\tilde{x}-x^*)^T(\tilde{x}-x^*) = \|\tilde{x}\|^2+\|x^*\|^2-2\tilde{x}^Tx = 2(1-\tilde{x}^Tx)$$ if we assume $$\|x^*\|=\|\tilde{x}\|=1$$. So, the absolute error is very related to $$1-\tilde{x}^Tx$$, which is a very similar error measure to what you showed: in this sense both error measures are equivalent. However, note that we started by using the standard Euclidean norm $$\|\bullet\|$$ to measure the absolute error. When we have a covariance matrix, it may be advantageous to use a Mahalanobis norm instead $$\|x\|_{\Sigma}^2 = x^T\Sigma^{-1}x$$, which gives "correct" weighting to individual parts of the error vector, according to their confidence. In this context, two things happen. First, under this new norm, the absolute error $$\|\tilde{x}-x^*\|^2$$ (using $$\Sigma = I$$) doesn't make a lot of sense for measuring error (hence, equivalently for your sine error), or would be very hard to relate to other quadratic expressions using another matrix $$\Sigma$$. Second, if we use an absolute error under the Mahalanobis distance $$\|\tilde{x}-x^*\|^2_{\Sigma} = \tilde{x}^T\Sigma^{-1}\tilde{x} + (x^*)^T\Sigma^{-1}x^* - 2\tilde{x}^T\Sigma^{-1}x^*$$ which do relates quadratic forms in $$\tilde{x}, x^{*}$$ and the matrix $$\Sigma$$. Perhaps in your problem it makes sense to use a Mahalanobis distance $$\|\bullet\|_{\Sigma^{-1}}$$ instead, so that you would obtain a clear relation between the error measure and the objective function $$\tilde{x}^T\Sigma\tilde{x}$$.

Aside from this, I don't know what else can you say about these expressions. Hope you find a better answer!