5
$\begingroup$

This question just occurred to me out of the blue. PCA is a way to reduce dimensions. Another way that is often (perhaps too often) used is to take mean values of two or more variables. This is done a great deal to create "scales" or "ratings" that get published in the popular press about various political issues. Dimensions are naively "averaged" and this cumulative "average" is portrayed as a magical one-number-sums-all figure.

Is this, nevertheless, a very crude relative of PCA? If so, what assumptions does it have? I can think of the following:

  • Sphericity is perfect or near-perfect.
  • All variables are perfectly orthogonal from all other variables.
  • Variables are all perfectly scaled against each other.

What else would there be?

$\endgroup$
  • $\begingroup$ I don't think this helps much. One way to think about it is that PCA is usually based on a correlation or covariance matrix, so means are subtracted out at the first step. So, the means of the variables are extra information, not something produced by PCA. There is a loose sense in which most statistics is a kind of averaging, which may be the direction of your thinking. Furthermore, a key feature of PCA is that the PCs are new variables, but if you reduce those to their means, you get zeros everywhere. $\endgroup$ – Nick Cox Apr 15 '14 at 17:58
  • 2
    $\begingroup$ Arguably this is not a question about data analysis at all, so PCA is practically irrelevant. It is a question about valuation. Because the scores are intended to be used to compare objects, the weights given to the variables and how these weighted values are combined determine how much of one variable can be traded off for how much of another variable. This is the subject of multi-attribute valuation theory, as discussed in a concrete version of the same question at stats.stackexchange.com/questions/9358. (One comment thread there also discusses the inapplicability of PCA.) $\endgroup$ – whuber Apr 15 '14 at 20:37
9
$\begingroup$

PCA forms linear combinations of the variables, and averaging all the variables is also taking a linear combination -- namely one where all the weights are equal to $1/d$, where $d$ is the number of variables. So one can view these approaches as conceptually related.

Moreover, under certain conditions averaging can indeed be called "a very crude relative of PCA", in a sense that PCA will result in the first principal component being proportional to the average of all variables (or close to it). What are these conditions?

Sphericity is perfect or near-perfect. All variables are perfectly orthogonal from all other variables. Variables are all perfectly scaled against each other.

  • If all variables are "perfectly scaled", let's assume that they are centread and standardized to have variances equal to $1$. This means that covariance matrix and correlation matrix coincide.

  • Note that if all the variables are indeed "perfectly orthogonal" to each other as you suggest, then the covariance/correlation matrix becomes identity matrix, and any vector can be chose to represent its first principal component; all eigenvalues are equal to $1$ and PCA would be useless (as would be the averaging). So let's rather consider small but non-zero pairwise covariances/correlations.

Now if all pairwise correlations are equal to the same number $c$, i.e. the covariance matrix looks like that: $$\left(\begin{array}{}1&c&c&c\\c&1&c&c\\c&c&1&c\\c&c&c&1\end{array} \right),$$ then the first eigenvector will be proportional to $$\left(\begin{array}{}1\\1\\1\\1 \end{array}\right),$$ i.e. the first PC will be proportional to the average over all variables. This should be obvious from the permutation-invariance of this covariance matrix.

This is true with any number of variables, not necessarily four. This also remains true for any value of $c \in (0,1)$, whether the variables are nearly orthogonal ($c\approx 0$) or not.

Moreover, this often remains approximately true (as noted by @whuber in the comments) if the off-diagonal elements are not exactly equal, but are of similar magnitude. Then the first PC will often be close to the average as well. For a nice real-life example of such a situation, see this answer illustrating the dataset of crab body measurements.

$\endgroup$
  • $\begingroup$ Need $c$ necessarily be small for this to hold? $\endgroup$ – Glen_b -Reinstate Monica Dec 29 '14 at 2:21
  • 1
    $\begingroup$ @Glen_b, no this certainly remains true with any $c$. I wrote about small one, because OP was going to assume "perfectly orthogonal" variables ($c=0$) and I suggested to consider $c \approx 0$ instead. Will edit to clarify. $\endgroup$ – amoeba says Reinstate Monica Dec 29 '14 at 10:11
  • 1
    $\begingroup$ Lest anyone be left with the impression that all pairwise correlations must be equal for $(1,\ldots,1)^\prime$ to be an eigenvector, let me point out that's not the case. But you still can make further progress along the lines you point out: if a vector close to $(1,\ldots,1)^\prime$ is an eigenvector of a matrix $A$, then (applying the definition of eigenvector) you can conclude that, to a good approximation, the row sums of $A$ are all equal. The converse is true when those row sums aren't too small (i.e., the eigenvalue is not small). That gives some insight into the question. $\endgroup$ – whuber Dec 29 '14 at 14:54
  • $\begingroup$ @whuber, yes, thank you for this clarification. What I wrote was of course a sufficient but not a necessary condition. However, I think the main interest here is in discussing when $(1,...,1)^\top$ is the first eigenvector (i.e. the one with the largest eigenvalue) as opposed to any eigenvector. I have no idea how would one describe necessary AND sufficient conditions for that. So I thought that pointing out a simple sufficient case would already be interesting. $\endgroup$ – amoeba says Reinstate Monica Dec 29 '14 at 15:03
  • 1
    $\begingroup$ In practice, this situation often appears to arise when all the coefficients of the correlation matrix are large in size. This occurs when the variables measure various aspects of a phenomenon which has a natural underlying magnitude, causing the values of all the variables to scale with that magnitude. Overall magnitude corresponds to the first PC. This will occur in testing situations and when making groups of related physical measurements. $\endgroup$ – whuber Dec 29 '14 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.