Do Principal Component Analysis (PCA) eliminate noise in the data set? If PCA do not eliminate noise in the data set, what actually does PCA do to the data set? Can somebody help me regarding this matter.
Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality.
It does not eliminate noise, but it can reduce noise.
Basically an orthogonal linear transformation is used to find a projection of all data into k dimensions, whereas these k dimensions are those of the highest variance. The eigenvectors of the covariance matrix (of the dataset) are the target dimensions and they can be ranked according to their eigenvalues. A high eigenvalue signifies high variance explained by the associated eigenvector dimension.
Lets take a look at the usps dataset, obtained by scanning handwritten digits from envelopes by the U.S. Postal Service.
First, we compute the eigenvectors and eigenvalues of the covariance matrix and plot all eigenvalues descending. We can see that there a few eigenvalues which could be named principal components, since their eigenvalues are much higher than the rest.
Each eigenvector is a linear combination of original dimensions. Therefore, the eigenvector (in this case) is an image itself, which can be plotted.
For b) dimensionality reduction, we could now use the top five eigenvectors and project all data (originally a 16*16 pixel image) into a 5 dimensional space with least possible loss of variance.
(Note here: In some cases, non-linear dimensionality reduction (such as LLE) might be better than PCA, see wikipedia for examples)
Finally we can use PCA for denoising. Therefore we can add extra noise to the original dataset in three levels (low, high, outlier) to be able to compare the performance. For this case I used gaussian noise with mean of zero and variance as a multiple of the original variance (Factor 1 (low), Factor 2 (high), Factor 20 (outlier) ) A possible result looks like this. Yet in each case, the parameter k must be tuned to find a good result.
Finally another perspective is to compare the eigenvalues of the highly noised data with the original data (compare with the first picture of this answer). You can see that the noise affects all eigenvalues, thus using only the top 25 eigenvalues for denoising, the influence of noise is reduced.