# Why do we want to maximize the variance in Principal Component Analysis?

I understand that in PCA, maximizing the variance is for preserving as much variability (or information) as possible during the process of reducing the dimension of the data", and I read the previous question, too: https://stackoverflow.com/questions/12395542/why-do-we-maximize-variance-during-principal-component-analysis

However, I still do not quite understand why we want to maximize the variance from the perspective of moments; for example, why don't we maximize higher-order moments, say, maximizing multiple even-order moments jointly (according to some desired weighting scheme), why just second moments?

• Good question. A somewhat flippant answer would be that if you were to (sequentially) maximize any other measure of spread, you wouldn't be doing PCA. Lurking behind this is a deeper idea: many methods to analyze and reduce the dimensions of a multivariate dataset have been invented, each with their own purposes, mathematical and statistical properties, and intended applications. PCA could be considered one of them. This suggests interpreting your question as concerning what properties distinguish PCA from the rest. That turns our attention away from moments and towards other aspects of PCA.
– whuber
Commented Sep 28, 2017 at 16:36

One answer is that maximizing variance minimizes squared error – a perhaps more immediately plausible goal.

Assume we want to reduce the dimensionality of a number of data points $$\mathbf{x}_1,\cdots, \mathbf{x}_N$$ to 1 by projecting onto a unit vector $$\mathbf{v}$$, and we want to keep the squared error small:

$$\underset{\mathbf{v}}{\text{minimize}} \, \sum_{n = 1}^N \left\|\mathbf{x}_n - (\mathbf{v}^\top \mathbf{x}_n)\mathbf{v}\right\|^2 \text{ subject to } \|\mathbf{v}\| = 1$$

This optimization problem can be turned into the equivalent problem

$$\underset{\mathbf{v}}{\text{maximize}} \,\, \mathbf{v}^\top \mathbf{C}\mathbf{v} \text{ subject to } \|\mathbf{v}\| = 1,$$ where $$\mathbf{C} = 1/N\sum_{n = 1}^N \mathbf{x}_n\mathbf{x}_n^\top$$. I.e., minimizing squared error is equivalent to maximizing variance along the direction of $$\mathbf{v}$$ (for centered data).

Another answer is that PCA is trying to fit a Gaussian model to the data (squared error and the Gaussian model are closely related). If you tried to fit another model to your data, you'd observe other moments as well (e.g., kurtosis becomes important when fitting a model via independent component analysis).

• important point: it minimizes squared error of reconstruction. That means that in terms of squared error, your PCA low-dimension vectors are the closest possible for that dimensionality to the true vectors. Commented Oct 5, 2017 at 13:10
• @bibliolytic closest possible -> no . closest w.r.t. to maximizing variance. There are other finding closest possible methods w.r.t other criteria. Commented Dec 22, 2020 at 2:47

Another answer is, "we really don't care at all about maximizing variance." After all, once we get the PCs, we multiply them by 10 if we like, we rotate them, etc. For example, if the PC coefficients are very similar, like .25, .30, .27, etc, we simply re-scale the coefficients so that they are close to 1.0 and call the PC a "summate." Clearly, this destroys the variance maximization subject to unit length constraint, calling into question whether variance maximization subject to unit length constraint has any relevance.

We provide an alternative to variance maximization in our article, "Teaching Principal Components Using Correlations," recently published in Multivariate Behavioral Research, https://www.ncbi.nlm.nih.gov/pubmed/28715259

Rather than define PCs as linear combinations that maximize variance, we summarize a (somewhat obscure) stream of literature that defines them as linear combinations that maximize average squared correlation between the linear combinations and the original variables. Then neither variance maximization nor the unit length constraint is needed. Re-scaling is allowed (encouraged even), and (non-singular) rotations are also allowed; all provide maximum average squared correlation with the original variables.

• Interesting paper! Your claim in the abstract that " the presentation can be made without matrix algebra or optimization proofs" seems to be because you don't prove your statement? Would you be able to prove that connection without matrix algebra or optimization proofs? Commented Apr 21, 2022 at 18:06
• Yes, the proofs are in the existing literature. You just have to dig to find them. The cites are given in the paper. Of course the approaches are equivalent. The point is that maximizing correlation is more intuitively related to the usual application of PC than is maximizing variance. Commented Aug 15, 2022 at 19:39

YAA (yet another answer). The first distribution we encounter is the normal distribution. In many important examples of statistics in use (linear regression, I'm looking at you) some hypothesis of normality is tacit. When statistics is taught, the normal distribution is the taught as the first and foremost example of a continuous distribution. The connection is that the normal distribution is completely characterized by it's mean and variance, which are it's first and second moment. So in any case of trying to understand some distribution in terms of it's moments, using any higher moment won't work sensibly if the normal distribution is among the possible outcomes.

• The normal distribution can be completely characterized in terms of its mean and fourth moment, too, so this argument doesn't seem particularly pertinent to the questions of "why variance" or "why maximize"?
– whuber
Commented Nov 19, 2017 at 16:33
• @whuber and presumably any two moments of different parity (via solving the equations of the moments explicitly). However I defy you to find any place in the literature etc where someone says something to the effect of consider the pdf = N(mean, fourth moment) etc. In essence, my answer is because it's the simplest way to do things.
– meh
Commented Nov 19, 2017 at 17:24
• "Simplest" in what way, exactly? I'm not objecting on the basis of simplicity, notation, or literature conventions. My objection is that the ideas you bring up ultimately have nothing essential to do with PCA at all--they're simply irrelevant to the ideas that underpin the concepts, theory, and (most) applications of PCA. BTW, it's standard not to parameterize Normal distributions with the mean and variance, but instead with $\mu/\sigma^2$ and $1/(2\sigma^2)$: see en.wikipedia.org/wiki/….
– whuber
Commented Nov 19, 2017 at 17:30
• Express variance in terms of the fourth moment please.
– meh
Commented Nov 19, 2017 at 21:06
• That's an illogical request. Again, you miss the point: this question explicitly asks, why not use the fourth moment? Indeed, the fourth moment could be substituted for the variance and one could produce a PCA-like algorithm based on it. Your answer does not effectively distinguish the fourth moment from the variance or the inverse of the variance or any other property that might be related to the spread of the data. The crucial concept, which has not emerged here, concerns how second moments change when data undergo linear transformations.
– whuber
Commented Nov 19, 2017 at 22:15