I have been reading and using tSNE which is able to preserve the neighbour around the a point in high dimension compared to PCA. For example I have these embeddings in 128 dimensional created by a neural network representing faces. Using both PCA, and tSNE I project them down into 2 dimensions.
It is evident that tSNE is doing a better job and in the paper for tSNE it states that:
For high-dimensional data that lies on or near a low-dimensional, non-linear manifold it is usually more important to keep the low-dimensional representations of very similar datapoints close together, which is typically not possible with a linear mapping.
Why is it typically not possible to keep very similar data points close together when using a linear mapping to map high-dimensional data to a low-dimension? I understand PCA so if possible would someone be able to provide a concrete example of why this is true?