5
$\begingroup$

I wanted to understand Singular Value Decomposition (SVD) hence consulted a few resources. In general I came across 2 different forms of SVD and hence got confused as to which one is correct or whether both are equivalent etc.

The 2 forms I came across are :

1) Page 45 of deep learning book (http://www.deeplearningbook.org/contents/linear_algebra.html)

The singular value decomposition is similar, except this time we will write A as a product of three matrices: $A = UDV^T$ ($^T$ implying transpose) Suppose that $A$ is an $m \times n$ matrix. Then $U$ is defined to be an $m \times m$ matrix, $D$ to be an $m \times n$ matrix, and $V$ to be an $n \times n$ matrix.

and

2) Lec 47 of Mining Massive Datasets course (https://www.youtube.com/watch?v=P5mlg91as1c&list=PLLssT5z_DsK9JDLcT8T62VtzwyW9LNepV&index=47)

$A = UDV^T$ ($^T$ implying transpose) Suppose that $A$ is an $m \times n$ matrix. Then $U$ is defined to be an $m \times r$ matrix, $D$ to be an $r \times r$ matrix, and $V$ to be an $n \times r$ matrix.

I consulted some more resources, it turns out that both the formulas are present in them as well. I need help to understand where is the gap in my understanding and what more do I need to cover (from a theoretical viewpoint) to bridge it.

$\endgroup$
5
$\begingroup$

I think that this question has been already covered on the site. I'm answering via an example. Both your citations are correct.

X
   -.910218691   -.037212328   -.275548166   -.360441580
  -1.855949714  -1.767057411   -.317283890   1.633887711
   -.186585570   -.316960442   -.340387838  -1.160337400
   1.427254188   -.872044568   -.447456104  -1.785203938
  -1.524117328   1.013061127    .735027680    .583921184
    .014211669   -.880275500   1.516716199   -.266348030

svd(x) = UDV'

D
   3.781503197    .000000000    .000000000    .000000000
    .000000000   2.427179503    .000000000    .000000000
    .000000000    .000000000   1.762938484    .000000000
    .000000000    .000000000    .000000000   1.494400129
    .000000000    .000000000    .000000000    .000000000
    .000000000    .000000000    .000000000    .000000000

U
  -.1045359966   .0613688431   .1533784675   .5925648676  -.7620680911  -.1729168539
  -.6655219390   .7063271894   .1384176823  -.0473139686   .1623946521  -.1020409758
   .1754941747   .1964628810   .0893095485   .6365658833   .3639134791   .6204665918
   .5934543082   .4282595430   .0031973635   .1744522991   .2294862700  -.6174932972
  -.4039262123  -.4669763619  -.2167568310   .4571230653   .4262068341  -.4256517430
  -.0031887094   .2393009491  -.9499187589   .0449114759  -.1616507584   .1105948958

V
   .7299152783  -.0317492361  -.0520390915  -.6808139286
   .0529861206  -.9763855955   .1901410033   .0878068268
  -.1023534922  -.1976805415  -.9745608685  -.0260245989
  -.6737506364  -.0811514539   .1066276016  -.7267083844

UDV' restores X
   -.910218691   -.037212328   -.275548166   -.360441580
  -1.855949714  -1.767057411   -.317283890   1.633887711
   -.186585570   -.316960442   -.340387838  -1.160337400
   1.427254188   -.872044568   -.447456104  -1.785203938
  -1.524117328   1.013061127    .735027680    .583921184
    .014211669   -.880275500   1.516716199   -.266348030

-----------------

Observe that (because X wasn't square) D above is not square.
The singular values are only four, the min(nrows,ncols) in X.
So, cut empty rows (or columns) in D so it becomes square, 4 x 4.
D
   3.781503197    .000000000    .000000000    .000000000
    .000000000   2.427179503    .000000000    .000000000
    .000000000    .000000000   1.762938484    .000000000
    .000000000    .000000000    .000000000   1.494400129

Then leave only 4 first columns ("real" left eigenvectors) in U
U
  -.1045359966   .0613688431   .1533784675   .5925648676
  -.6655219390   .7063271894   .1384176823  -.0473139686
   .1754941747   .1964628810   .0893095485   .6365658833
   .5934543082   .4282595430   .0031973635   .1744522991
  -.4039262123  -.4669763619  -.2167568310   .4571230653
  -.0031887094   .2393009491  -.9499187589   .0449114759

And only 4 first columns ("real" right eigenvectors) in V
(in this example, V was initially 4-column, so leave as is)
V
   .7299152783  -.0317492361  -.0520390915  -.6808139286
   .0529861206  -.9763855955   .1901410033   .0878068268
  -.1023534922  -.1976805415  -.9745608685  -.0260245989
  -.6737506364  -.0811514539   .1066276016  -.7267083844

UDV' restores X as well as it was above
   -.910218691   -.037212328   -.275548166   -.360441580
  -1.855949714  -1.767057411   -.317283890   1.633887711
   -.186585570   -.316960442   -.340387838  -1.160337400
   1.427254188   -.872044568   -.447456104  -1.785203938
  -1.524117328   1.013061127    .735027680    .583921184
    .014211669   -.880275500   1.516716199   -.266348030

Please note also a particular situation of singularity. If there are collinearities (co-proportionalities) in your matrix X one or more last singular values will be zero and D matrix might look like this, for example:

D: one zero singular value in 4-column [min(nrows,ncols) in X] D
   because of a singularity in X
   3.781503197    .000000000    .000000000    .000000000
    .000000000   2.427179503    .000000000    .000000000
    .000000000    .000000000   1.762938484    .000000000
    .000000000    .000000000    .000000000    .000000000
    .000000000    .000000000    .000000000    .000000000
    .000000000    .000000000    .000000000    .000000000

In such a case, you are in right to cut D to size 3 x 3 because only
3 singular values are "real" or tangible dimensions of the data.

Then, accordingly you will leave just 3 first columns in U and in V.
And you will successfully restore your X data with these truncated matrices.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.