I'm reading Data Driven Science and Engineering by Kutz and Brunton to understand more about the SVD.
Consider $X = U\Sigma V$, $XX^*$, and $X^*X$ where $X \in \mathbb{R}^{m\times n} $
In particular, consider that:
$XX^*U = U\Sigma^2$
and
$X^*XV = V\Sigma^2$
In the book, the authors mention that since the singular values are arranged in descending order by magnitude (in $\Sigma$), the columns of $U$ are ordered by how much correlation they capture in the columns of $X$, and the columns of $V$ are ordered by how much correlation they capture in the rows of $X$. I can't seem to understand why this is the case. In particular, I think it's the opposite: i.e that the columns of $U$ are ordered by how much correlation they capture in the rows of $X$, while the columns of $V$ are ordered by how much correlation they capture in the columns of $X$.
I think this is the case because when you compute $XX^*$, $XX^*_{i,j}$ corresponds to the dot product of the ith row of $X$ with the jth column of $X^*$ (i.e the jth row of $X$), and thus you are taking the dot product of two rows of $X$ (thus computing their correlation). This leads me to believe that $U$ has to do with correlation in the rows of $X$, not the columns. Similarly, when you compute $X^*X$, $X^*X_{i,j}$ takes the dot product of the ith row of $X^*$ (i.e the ith column of $X$) with the jth column of $X$, and thus you are taking the dot product of two columns of $X$ (thus computing their correlation). This again leads me to believe that $V$ has to do with correlation in the columns of $X$, not the rows.
I'd appreciate if someone could clarify.
Moreover, I'm trying to understand precisely how the fact that the columns of $U$ are hierarchically ordered (or $V$, but let's just consider $U$ for now) relates to the fact that each column is better able to capture correlation in $XX^*$ (i.e column 1 can better capture correlation than column 2, column 2 better than column 3 etc...). Here's how I currently understand it:
$$XX^* = U\Sigma^2 U^*$$ $$= \begin{bmatrix} \vert & & \vert \\ u_1 & ... & u_m \\ \vert & & \vert \end{bmatrix} \Sigma^2 \begin{bmatrix} \text{---} & u_1^T & \text{---} \\ & \text{...} \\ \text{---} & u_m^T & \text{---} \end{bmatrix} $$
We can write this as a dyadic summation (I believe this is what the following summation is called, I could be wrong though): $\sum_{i=1}^{m}\sigma_i\vec{u_i}\vec{u_i}^T$
Now, it is my hunch that for example $\sigma_1\vec{u_1}\vec{u_1}^T$ has a larger Frobenius norm than $\sigma_2\vec{u_2}\vec{u_2}^T$ because that would make it "closer" to $XX^*$, and thus better able to represent it (and hence better able to represent correlations). Actually as I'm writing this, I believe I found the answer to my second question, and convinced myself that this is indeed must be the case. Since the $\vec{u}$ vectors are orthonormal, and the singular values are hierarchically organized, it must be that $\sigma_1\vec{u_1}\vec{u_1}^T$ in the Frobenius sense is larger than $\sigma_2\vec{u_2}\vec{u_2}^T$. Since this has pretty much answered my second question, perhaps someone can add in additional insights!
