0
$\begingroup$

What's the expression for the variance of projection of $N$ data points onto a vector $\mathbf{v}$ which is not necessarily a unit vector? I could only find an expression when $\mathbf{v}$ is taken to be a unit vector.

Here's my attempt

Let's say our data is represented by a matrix $\mathbf{X} \in \mathbb{R}^{N \times p}$ and is already centered, i.e. $\mathbf{X} = \begin{bmatrix} (\mathbf{x}_1 - \bar{\mathbf{x}})^\intercal \\ \vdots \\ (\mathbf{x}_N - \bar{\mathbf{x}})^\intercal \end{bmatrix}$, where $\bar{\mathbf{x}} = \frac{1}{N} \sum_{i=1}^N \mathbf{x}_i$.

The projection for each data point is $\frac{\mathbf{x}_i^\intercal\mathbf{v}}{\mathbf{v}^\intercal\mathbf{v}}$. The variance can then be computed as follows:

$$ \begin{aligned} \hat{\sigma}^2 &= \frac{1}{N} \sum_{i=1}^N \Big(\frac{\mathbf{x}_i^\intercal\mathbf{v}}{\mathbf{v}^\intercal\mathbf{v}} - \frac{\bar{\mathbf{x}}^\intercal\mathbf{v}}{\mathbf{v}^\intercal\mathbf{v}} \Big)^2 \\ &= \frac{1}{N} \frac{1}{(\mathbf{v}^\intercal\mathbf{v})^2}\big(\mathbf{X}\mathbf{v}\big)^\intercal\big(\mathbf{X}\mathbf{v}\big) \\ &= \frac{1}{N} \frac{\mathbf{v}^\intercal\mathbf{X}^\intercal\mathbf{X}\mathbf{v}}{(\mathbf{v}^\intercal\mathbf{v})^2} \\ &= \frac{\mathbf{v}^\intercal \mathbf{S}\mathbf{v}}{(\mathbf{v}^\intercal\mathbf{v})^2} \end{aligned} $$

where $\mathbf{S}$ is the covariance matrix of the original data.

Is this correct? I am using it as an intermediate step in another problem and the expression I derived above for variance of projected data doesn't seem to be working.

$\endgroup$
1
$\begingroup$

The projection of $x$ onto the span of $v$ is a vector $\frac{x^\top v}{v^\top v} v$, but this vector has length $\frac{x^\top v}{v^\top v} \|v\| = \frac{x^\top v}{\sqrt{v^\top v}}$, which is what you should be using in your definition of variance. This will lead to $\frac{v^\top S v}{v^\top v}$ as the expression for variance (notice that replacing $v$ by $2v$ would not change this expression, but your expression $\frac{v^\top S v}{(v^\top v)^2}$ would get halved), and will lead to the desired result in your other post, as you noted in the comment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.