I understand that in PCA, maximizing the variance is for preserving as much variability (or information) as possible during the process of reducing the dimension of the data", and I read the previous question, too: https://stackoverflow.com/questions/12395542/why-do-we-maximize-variance-during-principal-component-analysis

However, I still do not quite understand why we want to maximize the variance from the perspective of moments; for example, why don't we maximize higher-order moments, say, maximizing multiple even-order moments jointly (according to some desired weighting scheme), why just second moments?

  • 1
    $\begingroup$ Good question. A somewhat flippant answer would be that if you were to (sequentially) maximize any other measure of spread, you wouldn't be doing PCA. Lurking behind this is a deeper idea: many methods to analyze and reduce the dimensions of a multivariate dataset have been invented, each with their own purposes, mathematical and statistical properties, and intended applications. PCA could be considered one of them. This suggests interpreting your question as concerning what properties distinguish PCA from the rest. That turns our attention away from moments and towards other aspects of PCA. $\endgroup$ – whuber Sep 28 '17 at 16:36

One answer is that maximizing variance minimizes squared error – a perhaps more immediately plausible goal.

Assume we want to reduce the dimensionality of a number of data points $\mathbf{x}_1, ..., \mathbf{x}_N$ to 1 by projecting onto a unit vector $\mathbf{v}$, and we want to keep the squared error small:

$$\underset{\mathbf{v}}{\text{minimize}} \, \sum_{n = 1}^N ||\mathbf{x}_n - (\mathbf{v}^\top \mathbf{x}_n)\mathbf{v}||^2 \text{ subject to } ||\mathbf{v}|| = 1$$

This optimization problem can be turned into the equivalent problem

$$\underset{\mathbf{v}}{\text{maximize}} \,\, \mathbf{v}^\top \mathbf{C}\mathbf{v} \text{ subject to } ||\mathbf{v}|| = 1,$$ where $\mathbf{C} = 1/N\sum_{n = 1}^N \mathbf{x}_n\mathbf{x}_n^\top$. I.e., minimizing squared error is equivalent to maximizing variance along the direction of $\mathbf{v}$ (for centered data).

Another answer is that PCA is trying to fit a Gaussian model to the data (squared error and the Gaussian model are closely related). If you tried to fit another model to your data, you'd observe other moments as well (e.g., kurtosis becomes important when fitting a model via independent component analysis).

  • 1
    $\begingroup$ important point: it minimizes squared error of reconstruction. That means that in terms of squared error, your PCA low-dimension vectors are the closest possible for that dimensionality to the true vectors. $\endgroup$ – bibliolytic Oct 5 '17 at 13:10

Another answer is, "we really don't care at all about maximizing variance." After all, once we get the PCs, we multiply them by 10 if we like, we rotate them, etc. For example, if the PC coefficients are very similar, like .25, .30, .27, etc, we simply re-scale the coefficients so that they are close to 1.0 and call the PC a "summate." Clearly, this destroys the variance maximization subject to unit length constraint, calling into question whether variance maximization subject to unit length constraint has any relevance.

We provide an alternative to variance maximization in our article, "Teaching Principal Components Using Correlations," recently published in Multivariate Behavioral Research, https://www.ncbi.nlm.nih.gov/pubmed/28715259

Rather than define PCs as linear combinations that maximize variance, we summarize a (somewhat obscure) stream of literature that defines them as linear combinations that maximize average squared correlation between the linear combinations and the original variables. Then neither variance maximization nor the unit length constraint is needed. Re-scaling is allowed (encouraged even), and (non-singular) rotations are also allowed; all provide maximum average squared correlation with the original variables.


YAA (yet another answer). The first distribution we encounter is the normal distribution. In many important examples of statistics in use (linear regression, I'm looking at you) some hypothesis of normality is tacit. When statistics is taught, the normal distribution is the taught as the first and foremost example of a continuous distribution. The connection is that the normal distribution is completely characterized by it's mean and variance, which are it's first and second moment. So in any case of trying to understand some distribution in terms of it's moments, using any higher moment won't work sensibly if the normal distribution is among the possible outcomes.

  • $\begingroup$ The normal distribution can be completely characterized in terms of its mean and fourth moment, too, so this argument doesn't seem particularly pertinent to the questions of "why variance" or "why maximize"? $\endgroup$ – whuber Nov 19 '17 at 16:33
  • $\begingroup$ @whuber and presumably any two moments of different parity (via solving the equations of the moments explicitly). However I defy you to find any place in the literature etc where someone says something to the effect of consider the pdf = N(mean, fourth moment) etc. In essence, my answer is because it's the simplest way to do things. $\endgroup$ – aginensky Nov 19 '17 at 17:24
  • $\begingroup$ "Simplest" in what way, exactly? I'm not objecting on the basis of simplicity, notation, or literature conventions. My objection is that the ideas you bring up ultimately have nothing essential to do with PCA at all--they're simply irrelevant to the ideas that underpin the concepts, theory, and (most) applications of PCA. BTW, it's standard not to parameterize Normal distributions with the mean and variance, but instead with $\mu/\sigma^2$ and $1/(2\sigma^2)$: see en.wikipedia.org/wiki/…. $\endgroup$ – whuber Nov 19 '17 at 17:30
  • $\begingroup$ Express variance in terms of the fourth moment please. $\endgroup$ – aginensky Nov 19 '17 at 21:06
  • $\begingroup$ That's an illogical request. Again, you miss the point: this question explicitly asks, why not use the fourth moment? Indeed, the fourth moment could be substituted for the variance and one could produce a PCA-like algorithm based on it. Your answer does not effectively distinguish the fourth moment from the variance or the inverse of the variance or any other property that might be related to the spread of the data. The crucial concept, which has not emerged here, concerns how second moments change when data undergo linear transformations. $\endgroup$ – whuber Nov 19 '17 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.