3
$\begingroup$

My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data points, or allows us to represent the data with using fewer dimensions.

My understanding is also closely tied to the visual image of PCA (or a non-linear manifold learning method) applied to noisy 2D data, the data being projected onto a learnt line where after a reconstruction is possible with reduced noise. (Example from answer by amoeba).

Noise reduction by PCA

I want to understand if there is a relationship between a noise reduction and dimension reduction. It seems to me that noise reduction will only be effective if the appropriate (In a bias-variance sense) model is used, although (potentially irresponsible) dimension reduction would be possible regardless of the model used.

  • What is the distinction between noise reduction and dimension reduction? Do these words belong in the same sentence?
  • Which fundamental mechanism in dimension reduction allows it to be capable of reducing the noise in data? What are the assumptions and trade-offs?
  • Apart from an independent test set, is there a way to gauge whether dimension reduction is discarding noise, or discarding information about the true distribution of the data?

Any guidance or suggested improvements to the question will be appreciated.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

What you see on the plot is not exactly a noise reduction. What is shown is the predictions made using only the first component of PCA. Using the first component of PCA leads to reducing multi-dimensional data to a single dimension. The new representation is less rich in information because to create it we disregarded the information that was represented in the other components. This can lead to noise reduction, but this also removes the non-noise, relevant information.

Moreover, it doesn't necessarily need to remove the noise. As an example, let's use a thought experiment: if you used PCA on completely "random" data (noise-only), the first component (or components) would contain something, as the algorithm forces them to be something. The algorithm in such a case wouldn't do anything to remove the noise, it would rather "overfit" to some bogus patterns.

Also keep in mind that what you see for PCA, or other dimensionality reduction algorithms, would also hold for any other machine learning algorithm that doesn't completely overfit the data. If the predictions by the machine learning model won't perfectly fit the data (they usually won't), it would produce predictions that are "simpler" and "smoother" than the observed data, because the algorithm needed internally to create a simplified representation of the data (it has fewer parameters than data points). Again, this doesn't necessarily remove the actual noise and whatever algorithm you would use for noise reduction, this is something to be verified when validating the model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Tim, thank you for the answer and useful thought experiment. Could we therefore frame the machine learning problem in this context as: Finding a model that is so simple that it is incapable of making predictions as complicated as the noisy data, yet flexible enough to ensure we don't discard information about the underlying process together with the noise? (All the while keeping in mind that we might be throwing away "noise information" in certain parts of the domain, and "useful information" in other parts) $\endgroup$
    – Douw Marx
    Commented Jul 20, 2022 at 13:29
  • 1
    $\begingroup$ @DouwMarx every time you build a model you discard some information. Also "noise" is the information that we consider as irrelevant or something that we can't explain. If you have test data with typos, the typos would be noise in one context, but if you are making research on why people do typos, this is valid data. There is no one "information" and one "noise". $\endgroup$
    – Tim
    Commented Jul 20, 2022 at 13:37
0
$\begingroup$

This is an awesome question. So the example of PCA that you have shown projects a set of points to a vector which is along the direction of maximum covariance of the data. What is does is that,

  1. It reduces one dimension from the data, but wait what just happened due to that ??...You lost an essential component of your representation and that may just look like you got rid of noise !! Is it ?

  2. The principal component direction along which you are projecting only reduces the error, which is the distance of the old point and the new projected point. (Check the derivation of PCA if you did not get this)

What it does not do is that,

  1. Hey that was not only noise that you got rid from, in the real world dataset if you want want to reduce dimensions and hope that you are getting rid of noise, you are, only if you consider the definition of noise in a mathematical setting. If you think in a realistic sense, the loss of representation (as I explained in the previous point) also takes place and that combined with loss of noise together cancels each other out in terms of actual benefit. The only time you benefit from that is when you do not need so many dimensions to represent the data.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.