14
$\begingroup$

I asked a similar question in the past, but I've thought about the message I am trying to convey a bit more and feel I can articulate it better. For context, I am on an introductory course in machine learning. A while ago we covered PCA, and from my point of view, I really can't see what it's good for in the real-world, and Google searching doesn't seem to shed much light on my question.

To illustrate my confusion, imagine we have a big data set and we would like to run PCA on it. Suppose (for convenience) that using 2 principal components explains an adequate amount of the variability. Now what? All we have information on now is the linear-combinations of a subset of the variables. I feel like at this point the data has become too abstracted to be interpretable in any meaningful way, in general. So, I reiterate, what's the point?

I might be missing something as PCA is used in industry all the time. My ignorance probably comes from only being on an introductory course. Does anyone have any perspective that might be useful?

$\endgroup$
4
  • $\begingroup$ If you've asked a similar question before, please give a cross-reference. $\endgroup$
    – Nick Cox
    Mar 18, 2023 at 10:22
  • 4
    $\begingroup$ My Google search turns up an incredible number of applications here on CV alone. $\endgroup$
    – whuber
    Mar 18, 2023 at 11:20
  • 3
    $\begingroup$ People use PCA in finance all the time to understand the risk factors in a portfolio. $\endgroup$
    – Wintermute
    Mar 18, 2023 at 15:07
  • $\begingroup$ A common example is face recognition (Although nowadays you would use neural networks of course): en.wikipedia.org/wiki/Eigenface $\endgroup$
    – KarlKastor
    Mar 18, 2023 at 20:58

8 Answers 8

20
$\begingroup$

One important use of PCA is in analysis of electroencephalography (EEG) data. To measure an EEG, dozens of electrodes are attached to your scalp and measure electric currents in your brain, either at rest or while you perform some experimental task.

Of course, the measurements at neighboring electrodes are heavily correlated, because they are generated by activity at a specific region in the brain, which then creates electric currents that will be picked up by all electrodes in the vicinity. It's not easy to learn about what happens deep in your brain if all you have is measurements from your scalp, but for some reason, few people are fine with having deep electrodes driven into their brain.

One thus reduces the dimensionality of the problem using PCA, which in this particular application also has a temporal component. You are completely right that it is hard to actually interpret the principal components. However, over the decades a body of research has developed that lets us expect particular principal components loading on particular electrodes, with peaks at particular points in time, e.g., after being presented with a specific stimulus.

For instance, a long time ago I looked at the P300, an event-related potential that loads over the parietal lobe (there's your PCA) about 300 ms after presentation of a stimulus that requires some kind of decision. In this particular analysis, the experiment was about whether spider phobics and non-phobics reacted differently to drawings that could be interpreted as spiders. The (unconscious) decision whether a particular drawing "was" a spider elicited a P300, and that indeed differed between phobics and nonphobics. Using a PCA and analyzing the parietal principal component - instead of, e.g., a single specific electrode = allows reducing the noise in such a setting, by essentially averaging the signal from multiple electrodes.

$\endgroup$
1
9
$\begingroup$

First, from the perspective of education, PCA is a good entryway to the world of dimension-reduction techniques and associated methods. Whether we're talking ICA, non-negative matrix factorization, confirmatory factor analysis, partial-least squares, canonical correlation analysis... (you get the idea), understanding PCA gets you halfway there.

From a practical standpoint, there are lots of times when PCA can be sensibly used to combine variables - I think the other answers give good examples.

Now, I think what your question is really getting at is - what's the point of a naive application of PCA? What's the point of throwing PCA at a long list of variables and getting back a handful of components? Well, it's quite useful if all you care about is model prediction, and it's also quite handy when you have a lot of highly correlated variables in your data set. PCA is also very useful if you want to understand the structure of your data - for example running PCA on genetic data (millions of variables) tells you about broad trends in ancestry in your data set.

$\endgroup$
3
  • 1
    $\begingroup$ Millions of variables? $\endgroup$
    – Nick Cox
    Mar 18, 2023 at 9:33
  • 2
    $\begingroup$ @NickCox Depends on the type of genetic data a person is interested in, but for something like variable nucleotides (generally known as single nucleotide polymorphisms, or SNPs), there can be several million in something like the human genome, if someone were to do full-genome sequencing. $\endgroup$
    – anjama
    Mar 18, 2023 at 11:56
  • 2
    $\begingroup$ Yes, genome-wide data set of SNPs will have tens of millions of variables these days. It's typically estimated that there are approximately 1 million independent sources of variance in genome-wide SNP data of common variants (i.e., present in at least 1% of the population). This is where the genome-wide significance cutoff of p<5x10^-8 comes from (Bonferroni correction for a million tests). $\endgroup$
    – David B
    Mar 18, 2023 at 12:33
8
$\begingroup$

I use three examples in my lectures to illustrate what PCA can do (click the links for pointers to the slides). They're chosen to show how useful it is in general data science practice and how powerful it can be (especially given that it's just a linear transformation).

example 1: bone analysis

Imagine you're a paleontologist, and you find a shoulder bone. Because of years of training, you recognize immediately that this is a Hominid bone (which is very rare) and not a Chimp bone (which is common).

How do you report that fact? "It's a Hominid bone because I say so", isn't very scientific.

One way would be to use PCA. You measure a bunch of features on the bone and on some similar bones, and you plot the first two principal components. Here's one example of such a plot [1].

A PCA plot of some Hominid and Ape shoulder bones.

The plot very clearly shows what your trained eyes told you in the first place: the bones of Hominids cluster together, far away from the more common bones of humans, chimps and other apes.

You can even draw a line backwards through the known evolutionary path, to get a hypothesis where humans come from. Turns out chimps and bonobos are a better candidate than gorillas and gibbons.

example 2: DNA

Take about 1300 people in Europe, sequence their DNA, and check about half a million markers. This gives you a dataset with 1300 instances, and 500 thousand features. Apply PCA, and plot it by the first two principal components. Now color the points by where the subject is from. Here's the result [2].

A PCA analysis of gene data.

The plot reveals that the first two principal components provide a blurry picture of the geographical distibution.

I admit it's hard to say what this is useful for, but it certainly illustrates the power of the method.

example 3: eigenfaces

Take a set of images of people's faces and flatten then into high-dimensional vectors. Run PCA.

Again, you will see clusters for certain meaningful concepts. But here, we can do something else cool. We can take the n-th principal component, and nudge one of the examples in our data a little in that direction.

Here is the result for the first couple of principal components:

An illustration of Eigenfaces on the Olivetti dataset.

1 Fossil hominin shoulders support an African ape-like last common ancestor of humans and chimpanzees. Nathan M. Young, Terence D. Capellini, Neil T. Roach and Zeresenay Alemseged http://www.pnas.org/content/112/38/11829

2 Novembre, J., Johnson, T., Bryc, K., Kutalik, Z., Boyko, A. R., Auton, A., ... & Bustamante, C. D. (2008). Genes mirror geography within Europe. Nature, 456(7218), 98-101. https://www.nature.com/articles/nature07331

$\endgroup$
2
  • 2
    $\begingroup$ Note that the use of PCA in population genetic studies has been recently criticized for leading to biased findings nature.com/articles/s41598-022-14395-4 The author calls for a reevaluation of tens of thousands of studies. $\endgroup$
    – J-J-J
    Mar 19, 2023 at 11:15
  • 4
    $\begingroup$ @J-J-J I wonder whether we should exercise a bit of scepticism when we read that one person concludes "32,000-216,000 genetic studies should be reevaluated". Anyway, one more reference about PCA in genetics: McVean G (2009) A Genealogical Interpretation of Principal Components Analysis. PLOS Genetics 5(10): e1000686. doi.org/10.1371/journal.pgen.1000686 $\endgroup$
    – dipetkov
    Mar 19, 2023 at 11:46
5
$\begingroup$

One application of PCA that I have used a few times is the construction of social indicators. We use the projection of each observation (usually households) over a component axis (usually the first), and use it as an indicator for public policy.

This is possible because the surveys that we use are designed to capture that information. You can look online for "quality of life questionnaires".

Another thing to take into account is that PCA may not be the best method in many applications, but is used for "backward compatibility". If you change the method, you will be unable to compare with past measurements. And the result will not be useful for building public policy.

see: http://article.sapub.org/10.5923.j.statistics.20221203.03.html

$\endgroup$
4
$\begingroup$

It is absolutely not clear, obvious, or agreed-upon what "intelligence" is in humans (or, for that matter, in nonhuman animals). What is clear is that people's performance on a variety of tasks involving mental processes (a very general term) is highly correlated. If you speak five languages, then chances are that your performance on standard matrix tests is also above average.

The probably currently best accepted "theory of intelligence" essentially boils down to performing a PCA on the result of multiple tests of mental processes and calling the first principal component the g factor. After you have done this, you start arguing with other psychologists about what precisely you are measuring, whether there truly is an underlying trait "intelligence", and what later principal components are measuring.

This construction gets more interesting and relevant as artificial intelligence (or, cough, not) gets better and better.

$\endgroup$
3
$\begingroup$

I've used PCA in facial motion capture for real time animatronic control of the 'lots of dots on a face' variety.

I was able to find out which dots - which is to say regions of the face - encoded the most information in movement and emotive expression.

It's obvious to some where these may be, and there is a natural correlation between these areas and how much they move, but I wanted to confirm my intuition.

I could only track so many dots so with that information I was able to more efficiently place them around the face, with more density in areas that encoded the most useful 'perpendicular' data and more sparsely in those that only became relevant on their own merits occasionally.

This is grossly simplified, and there was a lot more to it (the eyes... another world going on there) and I'd probably use a NN or similar with no dots this time round, but PCA played an integral part of the learning.

$\endgroup$
3
$\begingroup$

PCA just comes down to using the eigendecomposition of the (empirical) covariance matrix of the data. The full eigendecomposition of the covariance matrix results in a set of eigenvectors and corresponding eigenvalues, which can be interpreted as the variance along these eigenvectors. Because these eigenvectors are orthogonal, they can be used to create a rotation matrix that rotates the data so that the basis aligns with the direction of maximum variance.

For some people, this may be easier to understand with a few lines of code:

import numpy as np

data = np.random.randn(256, 2)  # generate some random 2D data
covariance = np.cov(data, rowvar=False)  # compute covariance matrix
variances, rotation = np.linalg.eigh(covariance)  # eigendecomposition
pcs = (data - data.mean(0)) @ rotation  # compute principal components

In other words, principal components are just a rotation version of the (centred) data. This also means that the (centred) data can simply be reconstructed by rotating back the principal components: pcs @ rotation.T. As a result, a "full" PCA (not sure if this is the correct term) is perfectly reversible. This is how PCA is typically used in the context of pre-processing data. After all, this rotation should typically make it easier to find important features. Typically, the principal components are additionally whitened (scaled to unit variance) using the eigenvalues of the decomposition (variances in the code). It is also possible to whiten the data after the rotation and then rotate it back, which gives rise to ZCA pre-processing. I can highly recommend this answer to a related question for further reading (and some nice figures).

When using PCA for dimensionality reduction, you would only use a subset of the columns in the rotation matrix. This obviously leads to loss of information. However, the columns of the rotation matrix can effectively be to transform the data back to the original space. After all, using only a few columns corresponds to setting principal components to zero (i.e. dropping information). If the total variance corresponding to these dropped dimensions is low enough, a reasonable reconstruction (of the centred data) is typically possible.

Again, in code:

data = np.random.randn(256, 784)  # generate some random high-D data
covariance = np.cov(data, rowvar=False)  # compute covariance matrix
_, rotation = np.linalg.eigh(covariance)  # eigendecomposition
reduced_pcs = (data - data.mean(0)) @ rotation[:, -70]  # dimensionality reduction PCA
reconstruction = reduced_pcs @ rotation[:, -70:].T  # reconstruction of (centred) data

Note that the reconstruction of the random data in this snippet of code is not going to work well. However, you should get some reasonable results if you plug in e.g. some MNIST data.

TL;DR: PCA is often used to pre-process data (make it nicer to work with) and can actually often be transformed back to the original input space (i.e. is not necessarily abstract).

$\endgroup$
1
  • 2
    $\begingroup$ The "full PCA" corresponds to a (normalized) singular value decomposition - you can simply multiply the SVD matrices back together to reconstruct the original. $\endgroup$
    – Michael B
    Mar 19, 2023 at 2:23
1
$\begingroup$

Lots of what we're trying to get at with dimensionality reduction, whether linear or nonlinear, is abstraction of the data away from a raw space and towards a manifold or embedding in which complex phenomena can be described with fewer, more meaningful parameters. PCA is arguably the simplest of these techniques; it will help you reduce correlated variables to a single common dimension. Implicit linear correlations in your dataset aside, lots of things in the real world are expressed as linear combinations of a set of latent variables. For example, you can use SVD (a generalization of PCA) in natural language because documents that contain words like "cat" are also likely to contain words like "pet" in rough proportion. So in a bag-of-words or tf-idf model, the correlation gets called out.

Aside from dimensionality reduction (or rather why it works for dimensionality reduction), PCA/SVD attempt to maximally explain the overall variance of the data using fewer components. You can think of this as "stretching" each component apart from the others as much as possible while keeping the component itself coherent (and this is actually explicitly how more advanced dimensionality reduction techniques such as maximum variance unfolding work). That makes it very useful at teasing apart how important each linear factor is within a dataset. That turns out to be particularly useful in recommendation engines over graphs, like the one that won the Netflix prize.

If PCA isn't useful today, it isn't because it inherently lacks usefulness, but because we have more powerful nonlinear techniques that have supplanted it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.