I don't understand why reduction in dimension is important. What is the benefit of taking some data and reducing their dimension?
Singular value decomposition (SVD) is not the same as reducing the dimensionality of the data. It is a method of decomposing a matrix into other matrices that has lots of wonderful properties which I won't go into here. For more on SVD, see the Wikipedia page.
Reducing the dimensionality of your data is sometimes very useful. It may be that you have a lot more variables than observations; this is not uncommon in genomic work. It may be that we have several variables that are very highly correlated, e.g., when they are heavily influenced by a small number of underlying factors, and we wish to recover some approximation to the underlying factors. Dimensionality-reducing techniques such as principal component analysis, multidimensional scaling, and canonical variate analysis give us insights into the relationships between observations and/or variables that we might not be able to get any other way.
A concrete example: some years ago I was analyzing an employee satisfaction survey that had over 100 questions on it. Well, no manager is ever going to be able to look at 100+ questions worth of answers, even summarized, and do more than guess at what it all means, because who can tell how the answers are related and what is driving them, really? I performed a factor analysis on the data, for which I had over 10,000 observations, and came up with five very clear and readily interpretable factors which could be used to develop manager-specific scores (one for each factor) that would summarize the entirety of the 100+ question survey. A much better solution than the Excel spreadsheet dump that had been the prior method of reporting results!
Regarding your secont point of the question, benefits of dimensionality reduction for a data set may be:
- reduce the storage space needed
- speed up computation (for example in machine learning algorithms), less dimensions mean les computing, also less dimensions can allow usage of algorithms unfit for a large number of dimensions
- remove redundant features, for example no point in storing a terrain's size in both sq meters and sq miles (maybe data gathering was flawed)
- reducing a data's dimension to 2D or 3D may allow us to plot and visualize it, maybe observe patterns, give us insights
Other than that, beyond PCA, SVD's has many applications in Signals Processing, NLP and many more
It is often used in facial recognition algorithms, and I make frequent use of it in my day job as a hedge fund analyst.