I am trying to understand the differences between the linear dimensionality reduction methods (e.g., PCA) and the nonlinear ones (e.g., Isomap).
I cannot quite understand what the (non)linearity implies in this context. I read from Wikipedia that
By comparison, if PCA (a linear dimensionality reduction algorithm) is used to reduce this same dataset into two dimensions, the resulting values are not so well organized. This demonstrates that the high-dimensional vectors (each representing a letter 'A') that sample this manifold vary in a non-linear manner.
What does
the high-dimensional vectors (each representing a letter 'A') that sample this manifold vary in a non-linear manner.
mean? Or more broadly, how do I understand the (non)linearity in this context?