Do Principal Component Analysis (PCA) eliminate noise in the data set? If PCA do not eliminate noise in the data set, what actually does PCA do to the data set? Can somebody help me regarding this matter.
$\begingroup$
$\endgroup$
14
-
1$\begingroup$ No it doesn't eliminate "noise" (in the sense that noisy data will remain noisy). PCA is just a transformation of data. Each PCA component represents a linear combination of predictors. And the PCAs can be ordered by their Eigenvalue: in broader sense the bigger the Eigenvalue the more variance is covered. Hence, lossless transformation would be when you have as much PC's as dimensions. Now, when you only consider some PC's with large Ev then you neglect components that add little to variance in the data (but this is not "noise"). $\endgroup$ – Drey Nov 22 '16 at 9:13
-
2$\begingroup$ As @Drey already noted, low variance components need not be noise. You could have noise as the high variance component, too. $\endgroup$ – Richard Hardy Nov 22 '16 at 10:22
-
$\begingroup$ Thank You. Actually i did what @Drey mention in his comment, which i eliminate PC's with small Ev that i previously thought it was noise inside the data set. So if i want continue to eliminate the PC's with small Ev, and used it as input for regression model and it improve the performacr of regression model. Can i say PCA had make the data easy to be intepreted and make prediction become more accurate. $\endgroup$ – bbadyalina Nov 22 '16 at 10:35
-
$\begingroup$ @Richard Hardy if PCA doest not rove noise from the data, how the linear transformation improve the data set? I somehow confuse about this, because there are a lot of researcher used PCA hybrid with time series model which improve the performance of prediction compare to conventional time series model. Thank You for your reply. $\endgroup$ – bbadyalina Nov 22 '16 at 10:40
-
$\begingroup$ Neither the data is "easy" (it is a linear combination of features) nor will it be easy to interpret (interpretation of coefficients in regression model). But your predictions may become more accurate. Even more, your model may generalize well. $\endgroup$ – Drey Nov 22 '16 at 10:40
|
Show 9 more comments
$\begingroup$
$\endgroup$
3
Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality.
It does not eliminate noise, but it can reduce noise.
Basically an orthogonal linear transformation is used to find a projection of all data into k dimensions, whereas these k dimensions are those of the highest variance. The eigenvectors of the covariance matrix (of the dataset) are the target dimensions and they can be ranked according to their eigenvalues. A high eigenvalue signifies high variance explained by the associated eigenvector dimension.
Lets take a look at the usps dataset, obtained by scanning handwritten digits from envelopes by the U.S. Postal Service.
First, we compute the eigenvectors and eigenvalues of the covariance matrix and plot all eigenvalues descending. We can see that there a few eigenvalues which could be named principal components, since their eigenvalues are much higher than the rest.
Each eigenvector is a linear combination of original dimensions. Therefore, the eigenvector (in this case) is an image itself, which can be plotted.
For b) dimensionality reduction, we could now use the top five eigenvectors and project all data (originally a 16*16 pixel image) into a 5 dimensional space with least possible loss of variance.
(Note here: In some cases, non-linear dimensionality reduction (such as LLE) might be better than PCA, see wikipedia for examples)
Finally we can use PCA for denoising. Therefore we can add extra noise to the original dataset in three levels (low, high, outlier) to be able to compare the performance. For this case I used gaussian noise with mean of zero and variance as a multiple of the original variance (Factor 1 (low), Factor 2 (high), Factor 20 (outlier) )
A possible result looks like this. Yet in each case, the parameter k must be tuned to find a good result.
Finally another perspective is to compare the eigenvalues of the highly noised data with the original data (compare with the first picture of this answer). You can see that the noise affects all eigenvalues, thus using only the top 25 eigenvalues for denoising, the influence of noise is reduced.
-
$\begingroup$ these are just figures did you try finding SNR for them $\endgroup$ – Boris Jun 2 '18 at 6:34
-
$\begingroup$ No, i just used these figures to illustrate the connection between noise reduction and PCA for an example dataset. You are welcome to write an answer that adds a new perspective. $\endgroup$ – Nikolas Rieble Jun 2 '18 at 8:06
-
$\begingroup$ Hi Nikolas, your answer is awesome, +1. I just posted a question on math stack exchange, I was wondering if you could help in answering it? Mainly, I'm confused about WHY the eigenvectors of the covariance matrix of the original dataset turn out to be the directions of highest variance, and thus we want to project onto them...here, I'm linking the question: math.stackexchange.com/questions/3213775/… Thanks! $\endgroup$ – joshuaronis May 4 '19 at 21:33
$\begingroup$
$\endgroup$
PCA is not designed for noise removal purpose. It is designed to REDUCE DIMENSIONS. As a large number of features are difficult to handle. PCA just lets you to approximate your data. Think PCA as a tuning knob. You can smoothly decide how much approximation you want by tuning it and which is impossible to achieve if you work directly with original given features. Because you cannot directly decide which feature to keep and which feature to eliminate to approximate your data at a desired level. Because the original features have no order of their priority or usability on which you can decide on which one to keep and which one to eliminate. That's why PCA comes into the place.
The main difference between the original dimensions and principle components is that, if you are working with original dimensions, you can think that the dimensions were already available before any data points were even plotted. So what wrong can happen? The problem is after plotting the data points these points can be positioned randomly and they may not allow us to eliminate some of the original dimensions directly to approximate it. As depending on the positions of the data points in many cases none of the original feature dimensions may be able to capture good variations notably. So the situation will be like, to capture a decent amount of variations of data you may need to keep a large number of original dimensions. Which is not efficient.
So what's the remedy? One of the remedies is using PCA! In PCA we do the opposite. Here the data points are already there. Now we will be placing the new dimensions (principle components) one by one with the target of capturing most of the variations available at that stage which are not still captured by the previous principle component(s) we have already plotted. Hence, the first PC covers the maximum possible variations (variation is measured by variance) possible to be captured by a single PC. The second PC captures the variations of data less than the first PC and those variations were missed out by the first PC. The third PC again does less than the second PC and so on. So, these principle components are already sorted based on how useful they are or how much variance they can capture. Each of the principle components has two properties - eigenvector and eigenvalue. The measure of the captured variations is nothing but the eigenvalue of that PC and the direction of that PC is just the eigenvector of it. As PCs are also axes and so each of them must have a direction.
So in your case, as you have said in the comment that after applying PCA the performance has improved, this is just because when you are eliminating some of the PCs of lower variances i.e, of lower eigenvalues, this action may be helping the model to generalize well. Because PCs of higher eigenvalues are capturing the more generalized features. As you are taking more and more PCs, the specialized features are also being added. If you take all of them the 100% of the data-variations will be restored like the original dimensions. So removing removing some PCs with lower eigenvalues actually acting as some sort of regularization and your model is only learning the more general features and not being confused by very fine detail which are likely not the general properties of that class. This is how overfitting is being prevented upto a certain level.
But again this doesn't assure you that those very fine example-specific details are noise. Noise can be embedded even with other PCs as well. Because PCA doesn't know which is noise and which is information. As it is just a linear transformation. All the PC axes can be represented by some linear combinations of existing original dimensions. So based on the variation levels of different types noises they can be captured by different PCs. So it is not a guaranteed way to remove noise although noise may be reduced if the eliminated PCs are involved in capturing those noise. But with noise you may also loose information as well.
