Do Principal Component Analysis (PCA) eliminate noise in the data set? If PCA do not eliminate noise in the data set, what actually does PCA do to the data set? Can somebody help me regarding this matter.
9
$\begingroup$
$\endgroup$
-
1$\begingroup$ No it doesn't eliminate "noise" (in the sense that noisy data will remain noisy). PCA is just a transformation of data. Each PCA component represents a linear combination of predictors. And the PCAs can be ordered by their Eigenvalue: in broader sense the bigger the Eigenvalue the more variance is covered. Hence, lossless transformation would be when you have as much PC's as dimensions. Now, when you only consider some PC's with large Ev then you neglect components that add little to variance in the data (but this is not "noise"). $\endgroup$ – Drey Nov 22 '16 at 9:13
-
2$\begingroup$ As @Drey already noted, low variance components need not be noise. You could have noise as the high variance component, too. $\endgroup$ – Richard Hardy Nov 22 '16 at 10:22
-
$\begingroup$ Thank You. Actually i did what @Drey mention in his comment, which i eliminate PC's with small Ev that i previously thought it was noise inside the data set. So if i want continue to eliminate the PC's with small Ev, and used it as input for regression model and it improve the performacr of regression model. Can i say PCA had make the data easy to be intepreted and make prediction become more accurate. $\endgroup$ – bbadyalina Nov 22 '16 at 10:35
-
$\begingroup$ @Richard Hardy if PCA doest not rove noise from the data, how the linear transformation improve the data set? I somehow confuse about this, because there are a lot of researcher used PCA hybrid with time series model which improve the performance of prediction compare to conventional time series model. Thank You for your reply. $\endgroup$ – bbadyalina Nov 22 '16 at 10:40
-
$\begingroup$ Neither the data is "easy" (it is a linear combination of features) nor will it be easy to interpret (interpretation of coefficients in regression model). But your predictions may become more accurate. Even more, your model may generalize well. $\endgroup$ – Drey Nov 22 '16 at 10:40
12
$\begingroup$
$\endgroup$
Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality.
It does not eliminate noise, but it can reduce noise.
Basically an orthogonal linear transformation is used to find a projection of all data into k dimensions, whereas these k dimensions are those of the highest variance. The eigenvectors of the covariance matrix (of the dataset) are the target dimensions and they can be ranked according to their eigenvalues. A high eigenvalue signifies high variance explained by the associated eigenvector dimension.
Lets take a look at the usps dataset, obtained by scanning handwritten digits from envelopes by the U.S. Postal Service.
First, we compute the eigenvectors and eigenvalues of the covariance matrix and plot all eigenvalues descending. We can see that there a few eigenvalues which could be named principal components, since their eigenvalues are much higher than the rest.
Each eigenvector is a linear combination of original dimensions. Therefore, the eigenvector (in this case) is an image itself, which can be plotted.
For b) dimensionality reduction, we could now use the top five eigenvectors and project all data (originally a 16*16 pixel image) into a 5 dimensional space with least possible loss of variance.
(Note here: In some cases, non-linear dimensionality reduction (such as LLE) might be better than PCA, see wikipedia for examples)
Finally we can use PCA for denoising. Therefore we can add extra noise to the original dataset in three levels (low, high, outlier) to be able to compare the performance. For this case I used gaussian noise with mean of zero and variance as a multiple of the original variance (Factor 1 (low), Factor 2 (high), Factor 20 (outlier) )
A possible result looks like this. Yet in each case, the parameter k must be tuned to find a good result.
Finally another perspective is to compare the eigenvalues of the highly noised data with the original data (compare with the first picture of this answer). You can see that the noise affects all eigenvalues, thus using only the top 25 eigenvalues for denoising, the influence of noise is reduced.
-
$\begingroup$ these are just figures did you try finding SNR for them $\endgroup$ – Boris Jun 2 '18 at 6:34
-
$\begingroup$ No, i just used these figures to illustrate the connection between noise reduction and PCA for an example dataset. You are welcome to write an answer that adds a new perspective. $\endgroup$ – Nikolas Rieble Jun 2 '18 at 8:06
-
$\begingroup$ Hi Nikolas, your answer is awesome, +1. I just posted a question on math stack exchange, I was wondering if you could help in answering it? Mainly, I'm confused about WHY the eigenvectors of the covariance matrix of the original dataset turn out to be the directions of highest variance, and thus we want to project onto them...here, I'm linking the question: math.stackexchange.com/questions/3213775/… Thanks! $\endgroup$ – Joshua Ronis May 4 at 21:33
