8
$\begingroup$

This question just occurred to me out of the blue. PCA is a way to reduce dimensions. Another way that is often (perhaps too often) used is to take mean values of two or more variables. This is done a great deal to create "scales" or "ratings" that get published in the popular press about various political issues. Dimensions are naively "averaged" and this cumulative "average" is portrayed as a magical one-number-sums-all figure.

Is this, nevertheless, a very crude relative of PCA? If so, what assumptions does it have? I can think of the following:

  • Sphericity is perfect or near-perfect.
  • All variables are perfectly orthogonal from all other variables.
  • Variables are all perfectly scaled against each other.

What else would there be?

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't think this helps much. One way to think about it is that PCA is usually based on a correlation or covariance matrix, so means are subtracted out at the first step. So, the means of the variables are extra information, not something produced by PCA. There is a loose sense in which most statistics is a kind of averaging, which may be the direction of your thinking. Furthermore, a key feature of PCA is that the PCs are new variables, but if you reduce those to their means, you get zeros everywhere. $\endgroup$
    – Nick Cox
    Commented Apr 15, 2014 at 17:58
  • 2
    $\begingroup$ Arguably this is not a question about data analysis at all, so PCA is practically irrelevant. It is a question about valuation. Because the scores are intended to be used to compare objects, the weights given to the variables and how these weighted values are combined determine how much of one variable can be traded off for how much of another variable. This is the subject of multi-attribute valuation theory, as discussed in a concrete version of the same question at stats.stackexchange.com/questions/9358. (One comment thread there also discusses the inapplicability of PCA.) $\endgroup$
    – whuber ♦
    Commented Apr 15, 2014 at 20:37

1 Answer 1

Reset to default
11
$\begingroup$

PCA forms linear combinations of the variables, and averaging all the variables is also taking a linear combination -- namely one where all the weights are equal to $1/d$, where $d$ is the number of variables. So one can view these approaches as conceptually related.

Moreover, under certain conditions averaging can indeed be called "a very crude relative of PCA", in a sense that PCA will result in the first principal component being proportional to the average of all variables (or close to it). What are these conditions?

Sphericity is perfect or near-perfect. All variables are perfectly orthogonal from all other variables. Variables are all perfectly scaled against each other.

  • If all variables are "perfectly scaled", let's assume that they are centread and standardized to have variances equal to $1$. This means that covariance matrix and correlation matrix coincide.

  • Note that if all the variables are indeed "perfectly orthogonal" to each other as you suggest, then the covariance/correlation matrix becomes identity matrix, and any vector can be chose to represent its first principal component; all eigenvalues are equal to $1$ and PCA would be useless (as would be the averaging). So let's rather consider small but non-zero pairwise covariances/correlations.

Now if all pairwise correlations are equal to the same number $c$, i.e. the covariance matrix looks like that: $$\left(\begin{array}{}1&c&c&c\\c&1&c&c\\c&c&1&c\\c&c&c&1\end{array} \right),$$ then the first eigenvector will be proportional to $$\left(\begin{array}{}1\\1\\1\\1 \end{array}\right),$$ i.e. the first PC will be proportional to the average over all variables. This should be obvious from the permutation-invariance of this covariance matrix.

This is true with any number of variables, not necessarily four. This also remains true for any value of $c \in (0,1)$, whether the variables are nearly orthogonal ($c\approx 0$) or not.

Moreover, this often remains approximately true (as noted by @whuber in the comments) if the off-diagonal elements are not exactly equal, but are of similar magnitude. Then the first PC will often be close to the average as well. For a nice real-life example of such a situation, see this answer illustrating the dataset of crab body measurements.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Need $c$ necessarily be small for this to hold? $\endgroup$
    – Glen_b
    Commented Dec 29, 2014 at 2:21
  • 1
    $\begingroup$ @Glen_b, no this certainly remains true with any $c$. I wrote about small one, because OP was going to assume "perfectly orthogonal" variables ($c=0$) and I suggested to consider $c \approx 0$ instead. Will edit to clarify. $\endgroup$
    – amoeba
    Commented Dec 29, 2014 at 10:11
  • 1
    $\begingroup$ Lest anyone be left with the impression that all pairwise correlations must be equal for $(1,\ldots,1)^\prime$ to be an eigenvector, let me point out that's not the case. But you still can make further progress along the lines you point out: if a vector close to $(1,\ldots,1)^\prime$ is an eigenvector of a matrix $A$, then (applying the definition of eigenvector) you can conclude that, to a good approximation, the row sums of $A$ are all equal. The converse is true when those row sums aren't too small (i.e., the eigenvalue is not small). That gives some insight into the question. $\endgroup$
    – whuber ♦
    Commented Dec 29, 2014 at 14:54
  • 2
    $\begingroup$ In practice, this situation often appears to arise when all the coefficients of the correlation matrix are large in size. This occurs when the variables measure various aspects of a phenomenon which has a natural underlying magnitude, causing the values of all the variables to scale with that magnitude. Overall magnitude corresponds to the first PC. This will occur in testing situations and when making groups of related physical measurements. $\endgroup$
    – whuber ♦
    Commented Dec 29, 2014 at 15:17
  • 2
    $\begingroup$ @whuber: one can look at it from the opposite direction and ask what the covariance matrix would look like if there is only one underlying factor with loadings $A = (a,...,a)^\top$, plus some noise $\Psi$ (assumed to be diagonal). The answer is of course $AA^\top + \Psi$, with all off-diagonal elements equal to $c = a^2$. If noise has the same variance for all variables, then PCA will exactly recover this factor. I think what you are saying is that off-diagonal elements can somewhat vary (instead of being constant), without much affecting the PC1. This is a good point. $\endgroup$
    – amoeba
    Commented Dec 29, 2014 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.