31
$\begingroup$

I learnt about PCA a few lectures ago in class and by digging more about this fascinating concept, I got to know about sparse PCA.

I wanted to ask, if I'm not wrong this is what sparse PCA is: In PCA, if you have $n$ data points with $p$ variables, you can represent each data point in $p$ dimensional space before applying PCA. After applying PCA, you can again represent it in the same dimensional space, but, this time, the first principal component will contain the most variance, the second will contain the second most variance direction and so on. So you can eliminate the last few principal components, as they will not cause a lot of loss of data, and you can compress the data. Right?

Sparse PCA is selecting principal components such that these components contain less non-zero values in their vector coefficients.

How is this supposed to help you interpret data better? Can anyone give an example?

$\endgroup$
4
  • 1
    $\begingroup$ Hello @GrowinMan! Have you seen my answer to this question? Do you think it answers it? If not, feel free to ask for any clarifications, or perhaps consider editing your question to make it more precise. If yes, then consider upvoting & "accepting" it by clicking a green tick nearby. I noticed that you have zero votes and zero accepted threads here on CrossValidated. $\endgroup$
    – amoeba
    Jul 26, 2016 at 13:33
  • $\begingroup$ @amoeba Thanks for pointing that out. Haven't logged in for a while and I'm also a little out of touch with machine learning. I'll be sure to read your answer again, and mark answers here by the weekend $\endgroup$
    – GrowinMan
    Aug 4, 2016 at 23:07
  • 1
    $\begingroup$ No problem. I have accidentally come across this old thread and thought of dropping you a line. $\endgroup$
    – amoeba
    Aug 4, 2016 at 23:14
  • 4
    $\begingroup$ Hello @GrowinMan! :-) Came across this old thread again. If you still feel this question is unresolved, please feel free to ask for clarifications. Otherwise, consider upvoting & "accepting" one of the answers by clicking a green tick nearby. I noticed that you have zero votes and zero accepted threads here on CrossValidated. $\endgroup$
    – amoeba
    Nov 7, 2018 at 22:41

4 Answers 4

43
$\begingroup$

Whether sparse PCA is easier to interpret than standard PCA or not, depends on the dataset you are investigating. Here is how I think about it: sometimes one is more interested in the PCA projections (low dimensional representation of the data), and sometimes -- in the principal axes; it is only in the latter case that sparse PCA can have any benefits for the interpretation. Let me give a couple of examples.

I am e.g. working with neural data (simultaneous recordings of many neurons) and am applying PCA and/or related dimensionality reduction techniques to get a low-dimensional representation of neural population activity. I might have 1000 neurons (i.e. my data live in 1000-dimensional space) and want to project it on the three leading principal axes. What these axes are, is totally irrelevant for me, and I have no intention of "interpreting" these axes in any way. What I am interested, is the 3D projection (as the activity depends on time, I get a trajectory in this 3D space). So I am fine if each axis has all 1000 non-zero coefficients.

On the other hand, somebody might be working with more "tangible" data, where individual dimensions have obvious meaning (unlike individual neurons above). E.g. a dataset of various cars, where dimensions are anything from weight to price. In this case one might actually be interested in the leading principal axes themselves, because one might want to say something: look, the 1st principal axis corresponds to the "fanciness" of the car (I am totally making this up now). If the projection is sparse, such interpretations would generally be easier to give, because many variables will have $0$ coefficients and so are obviously irrelevant for this particular axis. In the case of standard PCA, one usually gets non-zero coefficients for all variables.

You can find more examples and some discussion of the latter case in the 2006 Sparse PCA paper by Zou et al. The difference between the former and the latter case, however, I did not see explicitly discussed anywhere (even though it probably was).

$\endgroup$
2
  • 4
    $\begingroup$ This was a great explanation. Another example of your "tangible" data would be A survey with many questions and you want to know which questions on the survey are most important and perhaps some combination of them is really asking about one topic. $\endgroup$
    – bdeonovic
    Dec 12, 2013 at 0:53
  • 2
    $\begingroup$ Also, when PCA is part of a larger data-processing pipeline, sometimes there is a cost to collecting individual dimensions/variables. With standard PCA, even when you eliminate later PCs, you'll still need to collect all the same raw input variables because each of them is used in each PC. But by using sparse PCA instead to choose only a subset of the variables from your initial dataset, you may be able to save yourself time/money by not collecting the unused variables in the future. $\endgroup$
    – civilstat
    Dec 2, 2020 at 16:35
6
$\begingroup$

To understand the advantages of sparsity in PCA, you need to make sure you know the difference between "loadings" and "variables" (to me these names are somewhat arbitrary, but that's not important).

Say you have an $n\times p$ data matrix $\textbf{X}$, where $n$ is the number of samples. The SVD of $\textbf{X}=\textbf{US}\textbf{V}^\top$, gives you three matrices. Combining the first two $\textbf{Z} = \textbf{US}$ gives you the matrix of Principal Components. Let's say your reduced rank is $k$, then $\textbf{Z}$ is $n\times k$. $\textbf{Z}$ is essentially your data matrix after dimension reduction. Historically,

The entries of your principal components (aka $\textbf{Z} = \textbf{US}$) are called variables.

On the other hand, $\textbf{V}$ (which is $p\times k$) contains the Principal Loading Vectors and its entries are called the principal loadings. Given the properties of PCA, it's easy to show that $\textbf{Z}=\textbf{XV}$. This means that:

The principal components are derived by using the principal loadings as coefficients in a linear combination of your data matrix $\textbf{X}$.

Now that these definitions are out of the way, we'll look at sparsity. Most papers (or at least most that I've encountered), enforce sparsity on the principal loadings (aka $\textbf{V}$). The advantage of sparsity is that

a sparse $\textbf{V}$ will tell us which variables (from the original $p$-dimensional feature space) are worth keeping. This is called interpretability.

There are also interpretations for enforcing sparsity on the entries of $\textbf{Z}$, which I've seen people call "sparse variable PCA"", but that's far less popular and to be honest I haven't thought about it that much.

$\endgroup$
1
$\begingroup$

So you can eliminate the last few principal components, as they will not cause a lot of loss of data, and you can compress the data. Right?

yes, you're right. And if there are $N$ variables $V_1, V_2, \cdots , V_N$, you then have $N$ Principal Component $PC_1, PC_2, \cdots , PC_N$, and every variable $V_i$ has an information (a contribution) in every PC $PC_i$.

In Sparse PCA there are $PC_i$ without information of some variables $V_j, V_l, \cdots$, the variables with coefficient zero.

Then, if in one plane $(PC_i, PC_{j})$, there are fewer variables than expected ($N$), it's easier to clear the linear relations between them in this plane.

$\endgroup$
2
  • $\begingroup$ How!? I don't see how it would be any easy to interpret in this case as opposed to when the Principal Components are not sparse. $\endgroup$
    – GrowinMan
    Dec 10, 2013 at 23:38
  • 3
    $\begingroup$ The way I think about this is that we often do variable clustering before PC to make the results more interpretable. Sparse PC combines variable clustering and PC into one step, requiring fewer decisions on the part of the analyst. $\endgroup$ Oct 7, 2015 at 15:48
1
$\begingroup$

Like all good things, it depends.

After applying PCA, you can again represent it in the same dimensional space, but, this time, the first principal component will contain the most variance, the second will contain the second most variance direction and so on. So you can eliminate the last few principal components, as they will not cause a lot of loss of data, and you can compress the data. Right?

Yes. Here's the thing: the same logic applies for the parameters of each principal components themselves. Just as many of the principal components can be eliminated without losing very much information, you can eliminate many parts of each principal component without losing very much information. Why should we be efficient in one dimension but not another? Eliminate all of the unnecessary parameters.

If you think about the total number of parameters needed to describe your data, sparse PCA is very often just more efficient PCA. Modern sparse PCA methods are a big reason modern video and image compression is so good and doesn't look like sh-t. Sparse PCA is also easier to interpret; each principal component can actually represent a descriptive factor of some kind (for example: personality traits in psychological studies). And finally, sparse PCA (like sparse regression) is more likely to generalize better out of sample. If I have some new data generated from the same source, sparse principal components are more likely to summarize the new data better (less is more when it comes to generalizing).

Of course, that depends on your measure of efficiency. If you're using PCA to visualize multi-dimensional data in two or three dimensions, sparse PCA would be less efficient (each principal component would describe less variance than simple PCA).

Tl;dr simple PCA = handful of complicated principal components, sparse PCA = slightly more simple principal components and less parameters in total.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.