PCA variances pattern changed greatly after data cleaning

Question

I have data, when I normalize it and then performed PCA, I calculated the variance of PC components, I found that, the first component is 72% and seconed component is 8% (total 72+8=80%) and so on.

Now I used to the same data and performed some data cleaning function and then normalize it and then performed PCA, calculated variance of the PCA components I found that, first component gives 40% and second component gives 20% (40+20=60%).

What does this mean? Which one is better? I found that, after data cleaning operation, the classifier gives highest accuracy. I am unable to have a sense what is happening exactly and why? Would anyone explain?

Answer 1

This situation is typically conspicuous when you perform data cleaning, you must be clear about the data structure (highly balanced or unbalanced) before performing it. After all, it means that using the PCA before data cleaning tend to explain more variability of the data with less dimension.

To answer your question more precisely, could you tell me what classifier(s) are you using?

By the way, the formula of variance captured by component k is:

$$\frac{\lambda_k}{\sum_{i=1}^n \lambda_i}$$ ,where $\lambda_1 \ge \lambda_2 \ge......\ge \lambda_n \ge 0$ is the variance captured by each component.

Answer 2

For classifiers the critical issue is the ratio of variance within groups to variance between group means. With SVM the shape of the groups can be more complex than a simple linear discriminant analysis but the same principle holds (SVM can be viewed as remapping a complex group shape into a simple one).

What this means is that overall variance across the dataset can't predict performance in a classifier. If the data cleanup has removed irrelevant variations then the classifier should work better. If the variations removed by the cleanup are relevant to the classier it will be worse.