Do Principal Component Analysis (PCA) eliminate noise in the data set? If PCA do not eliminate noise in the data set, what actually does PCA do to the data set? Can somebody help me regarding this matter.
Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality.
It does not eliminate noise, but it can reduce noise.
Basically an orthogonal linear transformation is used to find a projection of all data into k dimensions, whereas these k dimensions are those of the highest variance. The eigenvectors of the covariance matrix (of the dataset) are the target dimensions and they can be ranked according to their eigenvalues. A high eigenvalue signifies high variance explained by the associated eigenvector dimension.
Lets take a look at the usps dataset, obtained by scanning handwritten digits from envelopes by the U.S. Postal Service.
First, we compute the eigenvectors and eigenvalues of the covariance matrix and plot all eigenvalues descending. We can see that there a few eigenvalues which could be named principal components, since their eigenvalues are much higher than the rest.
Each eigenvector is a linear combination of original dimensions. Therefore, the eigenvector (in this case) is an image itself, which can be plotted.
For b) dimensionality reduction, we could now use the top five eigenvectors and project all data (originally a 16*16 pixel image) into a 5 dimensional space with least possible loss of variance.
(Note here: In some cases, non-linear dimensionality reduction (such as LLE) might be better than PCA, see wikipedia for examples)
Finally we can use PCA for denoising. Therefore we can add extra noise to the original dataset in three levels (low, high, outlier) to be able to compare the performance. For this case I used gaussian noise with mean of zero and variance as a multiple of the original variance (Factor 1 (low), Factor 2 (high), Factor 20 (outlier) ) A possible result looks like this. Yet in each case, the parameter k must be tuned to find a good result.
Finally another perspective is to compare the eigenvalues of the highly noised data with the original data (compare with the first picture of this answer). You can see that the noise affects all eigenvalues, thus using only the top 25 eigenvalues for denoising, the influence of noise is reduced.
PCA is not designed for noise removal purpose. It is designed to REDUCE DIMENSIONS. As a large number of features are difficult to handle.
PCA just lets you to approximate your data. Think
PCA as a tuning knob. You can smoothly decide how much approximation you want by tuning it and which is impossible to achieve if you work directly with original given features. Because you cannot directly decide which feature to keep and which feature to eliminate to approximate your data at a desired level. Because the original features have no order of their priority or usability on which you can decide on which one to keep and which one to eliminate. That's why
PCA comes into the place.
The main difference between the original dimensions and principle components is that, if you are working with original dimensions, you can think that the dimensions were already available before any data points were even plotted. So what wrong can happen? The problem is after plotting the data points these points can be positioned randomly and they may not allow us to eliminate some of the original dimensions directly to approximate it. As depending on the positions of the data points in many cases none of the original feature dimensions may be able to capture good variations notably. So the situation will be like, to capture a decent amount of variations of data you may need to keep a large number of original dimensions. Which is not efficient.
So what's the remedy? One of the remedies is using
PCA we do the opposite. Here the data points are already there. Now we will be placing the new dimensions (principle components) one by one with the target of capturing most of the variations available at that stage which are not still captured by the previous principle component(s) we have already plotted. Hence, the
first PC covers the maximum possible variations (variation is measured by variance) possible to be captured by a single
second PC captures the variations of data less than the first
PC and those variations were missed out by the
first PC. The
third PC again does less than the
second PC and so on. So, these principle components are already sorted based on how useful they are or how much variance they can capture. Each of the principle components has two properties - eigenvector and eigenvalue. The measure of the captured variations is nothing but the eigenvalue of that
PC and the direction of that
PC is just the eigenvector of it. As
PCs are also axes and so each of them must have a direction.
So in your case, as you have said in the comment that after applying
PCA the performance has improved, this is just because when you are eliminating some of the
PCs of lower variances i.e, of lower eigenvalues, this action may be helping the model to generalize well. Because
PCs of higher eigenvalues are capturing the more generalized features. As you are taking more and more
PCs, the specialized features are also being added. If you take all of them the 100% of the data-variations will be restored like the original dimensions. So removing removing some
PCs with lower eigenvalues actually acting as some sort of regularization and your model is only learning the more general features and not being confused by very fine detail which are likely not the general properties of that class. This is how overfitting is being prevented upto a certain level.
But again this doesn't assure you that those very fine example-specific details are noise. Noise can be embedded even with other
PCs as well. Because
PCA doesn't know which is noise and which is information. As it is just a linear transformation. All the
PC axes can be represented by some linear combinations of existing original dimensions. So based on the variation levels of different types noises they can be captured by different
PCs. So it is not a guaranteed way to remove noise although noise may be reduced if the eliminated
PCs are involved in capturing those noise. But with noise you may also lose information as well.