# Measuring what's 'lost' in PCA dimensionality reduction?

I'm working with multiple time series signals ${\{X_i\}}$ and one method to remove suspected noise is with PCA. But unlike most methods which remove the components with least variance, we remove much more substantial components (often the second or third). This is often done because empirically it's been shown to improve results in certain analyses (like simple pairwise correlation) but has been detrimental to other analysis like clustering.

So I was curious of general metrics to compare before and after dimensionality reduction. Would looking at things like entropy be useful?

-
Any metric will almost invariably depend on the final "product" one is interested in. What are your end goals? –  cardinal Jan 26 '12 at 3:06
I can do empirical tests to show the impact on statistical thresholds (like t-tests) on pairwise correlation values and possibly clustering quality. But I wanted to know if there was a very general metric that would provide a potential indication on how methods I didn't test would perform. Ultimately, I would like to find out how detrimental is removing components that explain a large portion of the variance in the data to analysis. –  Swiss Army Man Jan 26 '12 at 3:25
It's not necessarily detrimental, though. I recall an interesting, and quite old, paper in the American Statistician in which the author shows several examples of the main effect being hidden in the smallest principal components. Unfortunately, I don't recall either the author or the date. It was probably late 1970s or early 1980s. –  cardinal Jan 26 '12 at 3:29
add comment

## 1 Answer

Conditional entropy or mutual information (between reduced and non-reduced data) would be in deed interesting metrics for the mentioned problem. But in order to apply those metrics you need to find an appropriate cluster/classification algorithm to convert to data into finite discrete domain. The clustering algorithm would determine the granularity of your data.

-
add comment