My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data points, or allows us to represent the data with using fewer dimensions.
My understanding is also closely tied to the visual image of PCA (or a non-linear manifold learning method) applied to noisy 2D data, the data being projected onto a learnt line where after a reconstruction is possible with reduced noise. (Example from answer by amoeba).
I want to understand if there is a relationship between a noise reduction and dimension reduction. It seems to me that noise reduction will only be effective if the appropriate (In a bias-variance sense) model is used, although (potentially irresponsible) dimension reduction would be possible regardless of the model used.
- What is the distinction between noise reduction and dimension reduction? Do these words belong in the same sentence?
- Which fundamental mechanism in dimension reduction allows it to be capable of reducing the noise in data? What are the assumptions and trade-offs?
- Apart from an independent test set, is there a way to gauge whether dimension reduction is discarding noise, or discarding information about the true distribution of the data?
Any guidance or suggested improvements to the question will be appreciated.