We start with an $m \times n$ matrix before SVD. After SVD, we have three matrices of sizes, $m \times m$, $n \times n$ and $m \times n$. How do we save space then if now we have three matrices instead of one and more numbers to store? Why are we talking here about dimensionality reduction? What are then the benefits of SVD?
For example, here's a $512 \times 512$ B&W image of Lena:
We compute the SVD of Lena. Choosing the singular values above $1\%$ of the maximum singular value, we are left with just $53$ singular values. Reconstructing Lena with these singular values and the corresponding (left and right) singular vectors, we obtain a low-rank approximation of Lena:
Instead of storing $512^2$ values (each taking $8$ bits), we can store $2 \cdot (512 \cdot 53) + 53 = 54325$ values, which is approximately $20\%$ of the original size. This is one example of how SVD can be used to do lossy compression.
Here's the MATLAB code:
% open Lena image and convert from uint8 to double Lena = double(imread('LenaBW.bmp')); % perform SVD on Lena [U,S,V] = svd(Lena); % extract singular values singvals = diag(S); % find out where to truncate the U, S, V matrices indices = find(singvals >= 0.01 * singvals(1)); % reduce SVD matrices U_red = U(:,indices); S_red = S(indices,indices); V_red = V(:,indices); % construct low-rank approximation of Lena Lena_red = U_red * S_red * V_red'; % print results to command window r = num2str(length(indices)); m = num2str(length(singvals)); disp(['Low-rank approximation used ',r,' of ',m,' singular values']); % save reduced Lena imwrite(uint8(Lena_red),'Reduced Lena.bmp');
We talk about dimensionality reduction when you use the SVD as part of Principal Component Analysis and choose to use a subset of the available dimensions. Check out: Relationship between SVD and PCA. How to use SVD to perform PCA?
SVD can be used for an incredible amount of things, few of which relate to reducing dimensionality.
Examples of use off the top of my head: calculating pseudoinverses, solving least squares problems, analysing time-dependent concentrations in chemical reactions, X-ray scattering time series analysis, non-linear corrections on pixel array detectors, estimating the rank/finding the null space of a matrix, and inversion problems.
SVD decomposes the X matrix to U, S and V matrices where:
U * S * V' = X with V being orthagonal
There are few ways of using that matrices to reduce dimension. One of the most common ways is to choose the first desired number of columns(vectors) from The V matrix which has n x n size. Then the dimension reduction takes place as you do this:
T = X * V_with_selected_vectors
T is generally called scores and V matrix represents loadings.
You can now use these scores with any kind of regression or classification. New samples must be projected to the same space using the same V. However, you should also take care of centering.
The main idea is projecting data in a way that the obtained scores explains the variance of the whole data in descending order. For instance, your first score vector may cover the 72% of the variance while the second score vector carries 12% of the variance and the third 5% so on... Thus, only a few number of principle components(the number of columns in score matrix) may account for almost all of the variance and since this score matrix has fewer variables(thus dimensions) many problems such as multicollinearity or computationally expensive tasks becomes easier to deal with.
One more thing to be noted is the explained variance is proportional with the corresponding eigenvalues that is contained in the S matrix with m x n size. For example, sum of the diagonal elements of S matrix divided by the square of the first element defines the explained variance of the first principle component.
There are many applications of SVD, the high multicollinearity of spectral data can be reduced using this method. Similarly, you can obtain black&white image from the RGB one by choosing a single principle component.