Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If you have a matrix with n rows and m columns, you can use SVD or other methods to calculate a low-rank approximation of the given matrix.

However, the low rank approximation will still have n rows and m columns. How can low-rank-approximations be useful for machine learning and natural language processing, given that you are left with the same number of features?

share|improve this question
They usually have sparse representations - you don't need to store $mn$ numbers for a low rank approximation. For example, a rank 1 approximation requires $n+m$ numbers. – probabilityislogic Oct 17 '13 at 7:26
up vote 14 down vote accepted

A low rank approximation $\hat{X}$ of $X$ can be decomposed into a matrix square root as $G=U_{r}\lambda_{r}^\frac{1}{2}$ where the eigen decomposition of $X$ is $U\lambda U^T$, thereby reducing the number of features, which can be represented by $G$ based on the rank-r approximation as $\hat{X}=GG^T$. Note that the subscript $r$ represents the number of eigen-vectors and eigen-values used in the approximation. Hence, it does reduce the number of features to represent the data. In some examples low-rank approximations are considered as basis or latent variable (dictionary) based expansions of the original data, under special constraints like orthogonality, non-negativity (non-negative matrix factorization) etc.

share|improve this answer

Once you have decided the rank of the approximation(say $r<m$) , you will only retain the $r$ basis vectors for future use (say, as predictors in a regression or classification problem) and not the original $m$.

share|improve this answer

The point of low-rank approximation is not necessarily for performing dimension reduction.

The idea is that based on domain knowledge, the data/entries of the matrix will make the matrix low rank. But that is in the ideal case where the entries are not affected by noise, corruption, missing values etc. The matrix in question typically will have much higher rank.

Low-rank approximation is thus a way to recover the "original" (the "ideal" matrix before it was messed up by noise etc.) low-rank matrix i.e., find the matrix that is most consistent (in terms of entries of the matrix) with the current matrix and is low-rank so that it can be used as an approximation to the "ideal" matrix. Having recovered this matrix, we can use it as a substitute for the noisy version and hopefully get better results.

share|improve this answer

Two more reasons not mentioned so far:

  1. Reducing colinearity. I believe that most of these techniques remove colinearity, which can be helpful for follow-on processing.

  2. Our imaginations are low-rank, so it can be helpful for exploring low-rank relationships.

share|improve this answer

According to "Modern multivariate statistical techniques (Izenman)" reduced rank regression covers several interesting methods as special cases including PCA, factor analysis, canonical variate and correlation analysis, LDA and correspondence analysis

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.