Let's say I have a $n \times m$ centered data matrix $A$ with SVD $A = U \Sigma V^{T}$.

For example, $m=50$ columns (measurements) that are spectra with $n=100$ different frequencies. The matrix is centered so the rows of the matrix have their mean subtracted. This is for interpreting the left singular vectors as principal components.

I'm interested in understanding how the SVD changes when I smooth each column vector. E.g. Let's smooth each 100x1 column by a simple kernel like [1/3, 2/3, 1/3].

$S=\begin{bmatrix} \frac{2}{3}&\frac{1}{3}& & & & &\\ \frac{1}{3}&\frac{2}{3}&\frac{1}{3} & & & &\\ & \frac{1}{3}&\frac{2}{3}&\frac{1}{3} & & &\\ & & \ddots& \ddots & \ddots & & \\ & & & \frac{1}{3}&\frac{2}{3}&\frac{1}{3} & \\ & & & &\frac{1}{3}&\frac{2}{3}&\frac{1}{3} \\ & & & & &\frac{1}{3}&\frac{2}{3} \end{bmatrix}$

So the new data matrix is $A'=SA$ and has some SVD $A' = U' \Sigma' V'^{T}$.

My first thought was that since $SA = SU \Sigma V^{T}$, I should expect the ${U'}_i$ vectors to be smoothed just like the $A_i$ vectors.

This first equation looks like the $(SU)_i$ are eigenvectors of $A' A'^T$:

$$SA(SA)^T=SU\Sigma\Sigma^TU^TS^T$$

But I'm not sure about this next one if its valid to say that the $V_i$ are eigenvectors of $A'^T A'$:

$$(SA)^TSA=V\Sigma^TU^TS^TSU\Sigma V^T$$

When I compare SVDs of $A$ and $A'$ numerically, the left singular vectors are not what I expect, $U' \neq SU$, but they do have a smoothed appearance.

Can anyone help me relate $U \Sigma V^{T}$ and the SVD $U' \Sigma' V'^{T}$ after this simple transformation?

up vote 8 down vote accepted

Why your first thoughts led you astray:

When you take the SVD of a matrix, $U$ and $V$ are unitary (orthogonal). So, while it is true that $SA = SU \Sigma V^{T}$, that is not (generally) the SVD of $SA$. Only if $S$ is unitary (which in the case of a smoothing matrix, it's not) would it be true that $U' = SU$.

Is there any elegant, symbolic way of relating the two SVDs? I can't find one. However, your smoothing matrix is a Toeplitz matrix. It's possible that such matrices have some special properties that might make for a more fruitful analysis. If you figure something out, please share with the rest of us.

The case of extreme smoothing:

One way to think about smoothing is a continuum from no smoothing to the extreme where we smooth each column to its mean value. Now, in that extreme case, the matrix would have a rank of 1, and there would only be one non-zero singular value. Let's look at the SVD:

$ \left[ \begin{matrix} \uparrow & \uparrow & & \uparrow \\ \mu_1 & \mu_2 & ... & \mu_m \\ \downarrow & \downarrow & & \downarrow \end{matrix} \right] = \left[ \begin{matrix} \boldsymbol{\mu} \\ \boldsymbol{\mu} \\ ... \\ \end{matrix} \right] = \mathbf{1} \boldsymbol{\mu}^T = \dfrac{\mathbf{1}}{\sqrt{n}} \left[ \|\boldsymbol{\mu}\| \sqrt{n} \right] \dfrac{\boldsymbol{\mu}^T}{\|\boldsymbol{\mu}\|} $

The last equation represents the truncated SVD. Note that the left and right vectors are of length 1. You can expand $\frac{\mathbf{1}}{\sqrt{n}}$ into an orthogonal matrix. Similarly for $\frac{\boldsymbol{\mu}}{\|\boldsymbol{\mu}\|}$. Then just zero-pad the middle matrix, and you've got the full SVD.

Intermediate smoothing

Presumably you're not going to do such extreme smoothing. So, what does this mean for you? As we broaden the smoothing, the spectrum gradually squishes down to a single value. For instance, in my simulations*:

Spectrum Normal Spectrum Ortho

As suggested by the derivation above, $U'_1$ will approach the normed 1-vector, and $V'_1$ will approach the normed mean-vector. But what about the other vectors?

As their corresponding singular values shrink, the other $U'_i$'s and $V'_i$'s will vary ever more wildly until they're just arbitrary choices for bases of the subspaces orthogonal to $U'_1$ and $V'_1$. That is to say, the'll just become noise.

If you need some intuition for why they're "just noise", consider that $SA$ is a weighted sum of dyads: $\displaystyle \sum \sigma_i U'_i V'^T_i $. We could completely change the directions of $U'_i$ and $V'_i$, and it will only affect the entries of $SA$ by less than $\sigma_i$.

Another visualization

Here's another way to look at column smoothing. Picture each row in the matrix as a point in $m$-space. As we smooth the columns, each point will get closer to the previous and next point. As a whole, the point cloud shrinks down†:

Row point cloud

Hope this helps!

[ * ]: I defined a family of increasingly broad smoothers. Roughly speaking, I took the kernel [1/4, 1/2, 1/4], convolved it $z$ times, clipped it to $d$ dimensions, and normalized so it summed to 1. Then I graphed the progressive smoothing of a random orthogonal and a random normal matrix.

[ † ]: Smoothers generated in the same way. $A$ is constructed as a series of points in $2$-space that look interesting.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.