I'm working with multiple time series signals ${\{X_i\}}$ and one method to remove suspected noise is with PCA. But unlike most methods which remove the components with least variance, we remove much more substantial components (often the second or third). This is often done because empirically it's been shown to improve results in certain analyses (like simple pairwise correlation) but has been detrimental to other analysis like clustering.

So I was curious of general metrics to compare before and after dimensionality reduction. Would looking at things like entropy be useful?

  • $\begingroup$ Any metric will almost invariably depend on the final "product" one is interested in. What are your end goals? $\endgroup$
    – cardinal
    Commented Jan 26, 2012 at 3:06
  • $\begingroup$ I can do empirical tests to show the impact on statistical thresholds (like t-tests) on pairwise correlation values and possibly clustering quality. But I wanted to know if there was a very general metric that would provide a potential indication on how methods I didn't test would perform. Ultimately, I would like to find out how detrimental is removing components that explain a large portion of the variance in the data to analysis. $\endgroup$ Commented Jan 26, 2012 at 3:25
  • 2
    $\begingroup$ It's not necessarily detrimental, though. I recall an interesting, and quite old, paper in the American Statistician in which the author shows several examples of the main effect being hidden in the smallest principal components. Unfortunately, I don't recall either the author or the date. It was probably late 1970s or early 1980s. $\endgroup$
    – cardinal
    Commented Jan 26, 2012 at 3:29
  • $\begingroup$ @cardinal I think I remember that paper as well. IIRC the goal was to show that the fundamentally linear nature of PCA means that non-linear relationships in the data will be missed. $\endgroup$ Commented Apr 9, 2015 at 2:33

1 Answer 1


Conditional entropy or mutual information (between reduced and non-reduced data) would be in deed interesting metrics for the mentioned problem. But in order to apply those metrics you need to find an appropriate cluster/classification algorithm to convert to data into finite discrete domain. The clustering algorithm would determine the granularity of your data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.