Motive behind preserving variance Dimensionality reduction techniques preserve some properties of the data. I was wondering how preserving variance (as PCA does) can be helpful?
Precisely speaking, PCA takes the covariance matrix and a number k, which denotes the dimensionality of the mapped space. It decomposes the co-variance matrix and selects the k-eigen vectors as the new dimension, such that the variance is preserved,as much as possible. I wanted to how preserving variance is useful? or in other words, what is the motive behind preserving variance?
 A: First of all, I'd say that there is no single dimensionality reduction technique that is always useful: PCA for dimensionality and/or noise reduction is a heuristic that is useful if you can sensibly assume that your data has roughly the following structure:
The data covers a not too large number ($n$) of "interesting" effects that cause relatively large variance, and in addition unrelated "noise" (and/or uninteresting influencing effects), which, however, has much lower magnitude (variance).
In such a situation, PCA will capture the interesting factors in the first few PCs which are kept for further analysis, while noise will tend to appear in the higher PCs and is removed.  
One conclusion from that is that PCA is e.g. not the method of choice if you know e.g. that certain confounding factors lead to large variance compared to the variation due to the effects of interest.
Also PCA produces linear combinations of the original variates. So if you'd e.g. expect your data to consist of few interesting channels among lots of noise-only channels then a dimensionality reduction based on hard feature selection could be more suitable. 
A: Suppose you deal with a linear regression $y=X\beta$, where a common estimator is $\hat\beta=(X'X)^{-1}y'X$. The variance (uncertainty) $\hat\sigma_\beta$ of the estimator $\hat\beta$ is inversely proportional to the variance of the data $\Omega_X=X'X/n$. 
So, if you replace the variables with PCA factors $F=AX$, then you would like to preserve the variance because: a) if you decrease the variance, then you increase the variance of the estimators $\hat\beta_F$, on the other hand b) if you increase the variance, it is only through artificial noise amplification, and the increased precision of the estimator $\beta_F$ is not real, i.e. you'll be underestimating the uncertainty of the coefficients.
A: Roughly speaking, variance is the amount of information contained in your data. The variables that have small variances provide little information about the data and can be removed from the analysis. For example, suppose you have a study of hospitalized dementia patients in which their demographics and health history were recorded. The patients may have different characteristics such as gender or different medication usage. But they are all elderly patients with age > 70. In this case, you cannot answer any question related to the age effect since all subjects are old. 
On the other hand, you may find 50% of your subjects are males. You should keep gender variable as it allows you to answer questions such as whether there is a difference between male and female patients.
