The SVD can be linked to dimensionality reduction from the standpoint of low rank matrix approximation.
SVD for low rank matrix approximation
Suppose we have a matrix $X$ and want to approximate it with a rank $r$ matrix $\hat{X}$, where $r < \text{rank}(X)$. The approximation error is typically measured by the Frobenius norm (which is equivalent to the square root of the squared error). The problem is then:
$$\min_\hat{X} \ \|X - \hat{X}\|_F \quad s.t. \ \text{rank}(\hat{X}) = r$$
The Eckart-Young theorem (Eckart and Young 1936) states that the solution is given by the truncated SVD of $X$:
$$\hat{X} = \tilde{U} \tilde{S} \tilde{V}^T$$
Where $X = U S V^T$ is the SVD of $X$, and $\tilde{U}, \tilde{S}, \tilde{V}$ are truncated versions of $U, S, V$--the bottom singular values and corresponding singular vectors have been discarded and only the top $r$ are retained.
Connection to dimensionality reduction
The rank of a data matrix indicates the number of dimensions spanned by the data points, i.e. the dimensionality of the linear (sub)space in which the points lie. If $X$ is a data matrix and $\hat{X}$ is a low rank approximation computed as above, this means that $\hat{X}$ is an approximation of $X$ where the points have been squashed into a lower dimensional subspace--specifically, an $r$-dimensional subspace. Hence, the truncated SVD performs dimensionality reduction.
Suppose rows correspond to data points and columns to dimensions. Then $\tilde{U} \tilde{S}$ (which has $r$ columns) gives low dimensional representations of the data. Multiplying by $\tilde{V}^T$ projects these low dimensional points back into the high dimensional space to approximate the original data: $X \approx \tilde{U} \tilde{S} \tilde{V}^T$
Connection to PCA
The relationship between SVD and PCA is often explained via eigendecomposition of the covariance matrix (e.g. as described here). But, there's also an alternative explanation based on the approximation error.
As above, the truncated SVD gives low dimensional representations that approximate the original data when projected back into the high dimensional space. This minimizes the approximation error as measured by the Frobenius norm, which is equivalent to minimizing the squared error. That is, it minimizes the squared distance between each data point and its reconstruction from the low dimensional representation.
This corresponds exactly to a description of PCA. PCA is commonly described as finding the directions of maximum variance, but it can be described equivalently as minimizing the squared approximation error, as above (e.g. see Tipping and Bishop 1999). Thus, SVD of the data matrix is equivalent to PCA. Note that this holds when the data matrix is centered. Otherwise, SVD corresponds to non-centered PCA (which can be expressed in terms of an eigendecomposition of $X^T X$ rather than the covariance matrix).
Other norms
Above, I focused on the Frobenius norm/squared error because that's the most common way of measuring reconstruction error. But, the truncated SVD also minimizes the reconstruction error as measured by other matrix norms. In particular, the Eckart-Young theorem can be generalized to all unitarily invariant norms (i.e. norms that are invariant to unitary transformations). See Mirsky (1960) and Li and Strang (2020). For example, unitarily invariant norms include the Ky Fan k-norms and Schatten p-norms, which include the common Frobenius, spectral, and nuclear norms as special cases.
References
Eckart and Young (1936). The approximation of one matrix by another of lower rank.
Li and Strang (2020). An elementary proof of Mirsky’s low rank approximation theorem.
Mirsky (1960). Symmetric gauge functions and unitarily invariant norms.
Tipping and Bishop (1999). Probabilistic principal component analysis.
