# How to use SVD for dimensionality reduction to reduce the number of columns (features) of the data matrix? [duplicate]

My original data has many more columns (features) than rows (users). I am trying to reduce the features of my SVD (I need all of the rows). I found one method of doing so in a book called "Machine Learning in Action" but I don't think it will work for the data I am using.

The method is as follows. Define SVD as $$A = USV^\top.$$

Set an optimization threshold (i.e., 90%). Calculate the total sum of the squares of the diagonal $S$ matrix. Calculate how many $S$ values it takes to reach 90% of the total sum of squares. So if that turns out to be 100 $S$ values, then I would take the first 100 columns of the $U$ matrix, first 100 rows of the $V^\top$ matrix, and a $100\times 100$ square matrix out of the $S$ matrix. I would then calculate $A = USV^\top$ using the reduced matrices.

However, this method does not target the columns of my original data, since the dimensions of the resulting $A$ matrix are the same as before. How would I target the columns of my original matrix?

## 2 Answers

What @davidhigh wrote is correct: if you multiply reduced versions of $\mathbf U_\mathrm{r}$, $\mathbf S_\mathrm{r}$, and $\mathbf V_\mathrm{r}$, as you describe in your question, then you will obtain a matrix $$\tilde{ \mathbf A}=\mathbf U_\mathrm{r}\mathbf S_\mathrm{r}\mathbf V_\mathrm{r}^\top$$ that has exactly the same dimensions as before, but has a reduced rank.

However, what @davidhigh did not add is that you can get what you want by multiplying reduced versions of $\mathbf U_\mathrm{r}$ and $\mathbf S_\mathrm{r}$ only, i.e. computing $$\mathbf B=\mathbf U_\mathrm{r}\mathbf S_\mathrm{r}.$$ This matrix has (in your example) only $100$ columns, but the same number of rows as $\mathbf A$. Matrix $\mathbf V$ is used only to map the data from this reduced 100-dimensional space to your original $p$-dimensional space. If you don't need to map it back, just leave $\mathbf V$ out, and done you are.

By the way, the columns of matrix $\mathbf B$ will contain what is called principal components of your data.

It seems that you are not completely aware of what an SVD does. As you wrote, it decomposes a matrix $\mathbf A$ according to

$$\mathbf A = \mathbf U \mathbf S \mathbf V^T,$$

Read the details on the involved matrix dimensions and properties for example here.

Now, dimensionality reduction is done by neglecting small singular values in the diagonal matrix $\mathbf S$. Regardless of how many singular values you approximately set to zero, the resulting matrix $\mathbf A$ always retains its original dimension. In particular, you don't drop any rows or columns.

Consequently, the feature of dimensionality reduction is only exploited in the decomposed version. Consider for example a very large matrix with rank 1, that is, the column/row-vectors span only a one-dimensional subspace. For this matrix, you will obtain only one non-zero singular value. Now, instead of storing this large matrix one can also store two vectors and one real number, which corresponds to a reduction by one order of magnitude.

• So if I understand correctly, SVD is only useful in reducing the storage used (while retaining properties of original matrix)? It doesn't actually reduce the dimensions of the original dataset (since multiplying USV^T gives you A with its original dimension) and the original dataset is what is needed to run similarity algorithms. – covfefe Jul 13 '14 at 4:28
• It depends on what you want to do with the SVD. If you still feed in the SVD-reduced matrix $\mathbf A$ into your algorithm, the effort remains the same, but results might be better as you concentrate on directions with maximum variance (and hopefully filtered out noise). However, it is also possible to directly put features of the decomposition into your algorithm (this, however, means that you need to apply a similar decomposition for each prediction). If reduction of the algorithmic effort is what you are aiming for, I would rather look for a dual version of your algorithm – davidhigh Jul 13 '14 at 10:00
• What do you mean by 'dual version'? I'm using the SVD on a matrix of documents and features and I'm hoping to reduce the number of features by eliminating insignificant ones. I want to use the reduced matrix to find similarities between documents. – covfefe Jul 13 '14 at 23:59