Confusion regarding the dimensions of matrices in the formula for SVD

Question

I wanted to understand Singular Value Decomposition (SVD) hence consulted a few resources. In general I came across 2 different forms of SVD and hence got confused as to which one is correct or whether both are equivalent etc.

The 2 forms I came across are :

1) Page 45 of deep learning book (http://www.deeplearningbook.org/contents/linear_algebra.html)

The singular value decomposition is similar, except this time we will write A as a product of three matrices: $A = UDV^T$ ($^T$ implying transpose) Suppose that $A$ is an $m \times n$ matrix. Then $U$ is defined to be an $m \times m$ matrix, $D$ to be an $m \times n$ matrix, and $V$ to be an $n \times n$ matrix.

and

2) Lec 47 of Mining Massive Datasets course (https://www.youtube.com/watch?v=P5mlg91as1c&list=PLLssT5z_DsK9JDLcT8T62VtzwyW9LNepV&index=47)

$A = UDV^T$ ($^T$ implying transpose) Suppose that $A$ is an $m \times n$ matrix. Then $U$ is defined to be an $m \times r$ matrix, $D$ to be an $r \times r$ matrix, and $V$ to be an $n \times r$ matrix.

I consulted some more resources, it turns out that both the formulas are present in them as well. I need help to understand where is the gap in my understanding and what more do I need to cover (from a theoretical viewpoint) to bridge it.

Answer 1

I think that this question has been already covered on the site. I'm answering via an example. Both your citations are correct.

X
   -.910218691   -.037212328   -.275548166   -.360441580
  -1.855949714  -1.767057411   -.317283890   1.633887711
   -.186585570   -.316960442   -.340387838  -1.160337400
   1.427254188   -.872044568   -.447456104  -1.785203938
  -1.524117328   1.013061127    .735027680    .583921184
    .014211669   -.880275500   1.516716199   -.266348030

svd(x) = UDV'

D
   3.781503197    .000000000    .000000000    .000000000
    .000000000   2.427179503    .000000000    .000000000
    .000000000    .000000000   1.762938484    .000000000
    .000000000    .000000000    .000000000   1.494400129
    .000000000    .000000000    .000000000    .000000000
    .000000000    .000000000    .000000000    .000000000

U
  -.1045359966   .0613688431   .1533784675   .5925648676  -.7620680911  -.1729168539
  -.6655219390   .7063271894   .1384176823  -.0473139686   .1623946521  -.1020409758
   .1754941747   .1964628810   .0893095485   .6365658833   .3639134791   .6204665918
   .5934543082   .4282595430   .0031973635   .1744522991   .2294862700  -.6174932972
  -.4039262123  -.4669763619  -.2167568310   .4571230653   .4262068341  -.4256517430
  -.0031887094   .2393009491  -.9499187589   .0449114759  -.1616507584   .1105948958

V
   .7299152783  -.0317492361  -.0520390915  -.6808139286
   .0529861206  -.9763855955   .1901410033   .0878068268
  -.1023534922  -.1976805415  -.9745608685  -.0260245989
  -.6737506364  -.0811514539   .1066276016  -.7267083844

UDV' restores X
   -.910218691   -.037212328   -.275548166   -.360441580
  -1.855949714  -1.767057411   -.317283890   1.633887711
   -.186585570   -.316960442   -.340387838  -1.160337400
   1.427254188   -.872044568   -.447456104  -1.785203938
  -1.524117328   1.013061127    .735027680    .583921184
    .014211669   -.880275500   1.516716199   -.266348030

-----------------

Observe that (because X wasn't square) D above is not square.
The singular values are only four, the min(nrows,ncols) in X.
So, cut empty rows (or columns) in D so it becomes square, 4 x 4.
D
   3.781503197    .000000000    .000000000    .000000000
    .000000000   2.427179503    .000000000    .000000000
    .000000000    .000000000   1.762938484    .000000000
    .000000000    .000000000    .000000000   1.494400129

Then leave only 4 first columns ("real" left eigenvectors) in U
U
  -.1045359966   .0613688431   .1533784675   .5925648676
  -.6655219390   .7063271894   .1384176823  -.0473139686
   .1754941747   .1964628810   .0893095485   .6365658833
   .5934543082   .4282595430   .0031973635   .1744522991
  -.4039262123  -.4669763619  -.2167568310   .4571230653
  -.0031887094   .2393009491  -.9499187589   .0449114759

And only 4 first columns ("real" right eigenvectors) in V
(in this example, V was initially 4-column, so leave as is)
V
   .7299152783  -.0317492361  -.0520390915  -.6808139286
   .0529861206  -.9763855955   .1901410033   .0878068268
  -.1023534922  -.1976805415  -.9745608685  -.0260245989
  -.6737506364  -.0811514539   .1066276016  -.7267083844

UDV' restores X as well as it was above
   -.910218691   -.037212328   -.275548166   -.360441580
  -1.855949714  -1.767057411   -.317283890   1.633887711
   -.186585570   -.316960442   -.340387838  -1.160337400
   1.427254188   -.872044568   -.447456104  -1.785203938
  -1.524117328   1.013061127    .735027680    .583921184
    .014211669   -.880275500   1.516716199   -.266348030

Please note also a particular situation of singularity. If there are collinearities (co-proportionalities) in your matrix X one or more last singular values will be zero and D matrix might look like this, for example:

D: one zero singular value in 4-column [min(nrows,ncols) in X] D
   because of a singularity in X
   3.781503197    .000000000    .000000000    .000000000
    .000000000   2.427179503    .000000000    .000000000
    .000000000    .000000000   1.762938484    .000000000
    .000000000    .000000000    .000000000    .000000000
    .000000000    .000000000    .000000000    .000000000
    .000000000    .000000000    .000000000    .000000000

In such a case, you are in right to cut D to size 3 x 3 because only
3 singular values are "real" or tangible dimensions of the data.

Then, accordingly you will leave just 3 first columns in U and in V.
And you will successfully restore your X data with these truncated matrices.