1
$\begingroup$

Dimensionality reduction techniques preserve some properties of the data. I was wondering how preserving variance (as PCA does) can be helpful?

Precisely speaking, PCA takes the covariance matrix and a number k, which denotes the dimensionality of the mapped space. It decomposes the co-variance matrix and selects the k-eigen vectors as the new dimension, such that the variance is preserved,as much as possible. I wanted to how preserving variance is useful? or in other words, what is the motive behind preserving variance?

$\endgroup$
  • 1
    $\begingroup$ This question needs some improvement: it is not at all clear what you are asking. You can edit your answer to improve it (and prevent it from being closed) by clicking the "edit" button in the lower left. $\endgroup$ – Alexis Oct 27 '14 at 5:56
  • $\begingroup$ This might be a mis-characterization of PCA. If one wanted truly to "preserve" variance, it would be a simple matter of multiplying the retained eigenvectors by a constant value (usually slightly greater than $1$). That this is not done signals that PCA has different aims. Perhaps you could clarify what you mean by "preserving variance"? $\endgroup$ – whuber Oct 28 '14 at 17:19
1
$\begingroup$

First of all, I'd say that there is no single dimensionality reduction technique that is always useful: PCA for dimensionality and/or noise reduction is a heuristic that is useful if you can sensibly assume that your data has roughly the following structure:

The data covers a not too large number ($n$) of "interesting" effects that cause relatively large variance, and in addition unrelated "noise" (and/or uninteresting influencing effects), which, however, has much lower magnitude (variance).

In such a situation, PCA will capture the interesting factors in the first few PCs which are kept for further analysis, while noise will tend to appear in the higher PCs and is removed.

One conclusion from that is that PCA is e.g. not the method of choice if you know e.g. that certain confounding factors lead to large variance compared to the variation due to the effects of interest.
Also PCA produces linear combinations of the original variates. So if you'd e.g. expect your data to consist of few interesting channels among lots of noise-only channels then a dimensionality reduction based on hard feature selection could be more suitable.

$\endgroup$
-1
$\begingroup$

Roughly speaking, variance is the amount of information contained in your data. The variables that have small variances provide little information about the data and can be removed from the analysis. For example, suppose you have a study of hospitalized dementia patients in which their demographics and health history were recorded. The patients may have different characteristics such as gender or different medication usage. But they are all elderly patients with age > 70. In this case, you cannot answer any question related to the age effect since all subjects are old.

On the other hand, you may find 50% of your subjects are males. You should keep gender variable as it allows you to answer questions such as whether there is a difference between male and female patients.

$\endgroup$
  • 1
    $\begingroup$ "Small" and "large" variances within "data" (or just the independent variables) are meaningless assessments. Among other things, the variance depends on the units of measurement, which in most applications are arbitrary. $\endgroup$ – whuber Oct 28 '14 at 21:31
  • $\begingroup$ @whuber I partially agree with you and this is one of the criticism of PCA. $\endgroup$ – Peter Oct 29 '14 at 10:53
  • $\begingroup$ (-1) Variance of each individual variable has nothing to do with the "amount of information" contained in it (especially in your example with age and gender), because it simply depends on the units of measurement, as @whuber wrote. $\endgroup$ – amoeba says Reinstate Monica Oct 30 '14 at 23:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.