How exactly is sparse PCA better than PCA? I learnt about PCA a few lectures ago in class and by digging more about this fascinating concept, I got to know about sparse PCA.
I wanted to ask, if I'm not wrong this is what sparse PCA is:
In PCA, if you have $n$ data points with $p$ variables, you can represent each data point in $p$ dimensional space before applying PCA. After applying PCA, you can again represent it in the same dimensional space, but, this time, the first principal component will contain the most variance, the second will contain the second most variance direction and so on. So you can eliminate the last few principal components, as they will not cause a lot of loss of data, and you can compress the data. Right?
Sparse PCA is selecting principal components such that these components contain less non-zero values in their vector coefficients. 
How is this supposed to help you interpret data better?
Can anyone give an example?
 A: To understand the advantages of sparsity in PCA, you need to make sure you know the difference between "loadings" and "variables" (to me these names are somewhat arbitrary, but that's not important). 
Say you have an $n\times p$ data matrix $\textbf{X}$, where $n$ is the number of samples. The SVD of $\textbf{X}=\textbf{US}\textbf{V}^\top$, gives you three matrices. Combining the first two $\textbf{Z} = \textbf{US}$ gives you the matrix of Principal Components. Let's say your reduced rank is $k$, then $\textbf{Z}$ is $n\times k$. $\textbf{Z}$ is essentially your data matrix after dimension reduction. Historically, 

The entries of your principal components (aka $\textbf{Z} = \textbf{US}$) are called variables. 

On the other hand, $\textbf{V}$ (which is $p\times k$) contains the Principal Loading Vectors and its entries are called the principal loadings. Given the properties of PCA, it's easy to show that $\textbf{Z}=\textbf{XV}$. This means that:

The principal components are derived by using the principal loadings as coefficients in a linear combination of your data matrix $\textbf{X}$. 

Now that these definitions are out of the way, we'll look at sparsity. Most papers (or at least most that I've encountered), enforce sparsity on the principal loadings (aka $\textbf{V}$). The advantage of sparsity is that 

a sparse $\textbf{V}$ will tell us which variables (from the original $p$-dimensional feature space) are worth keeping. This is called interpretability. 

There are also interpretations for enforcing sparsity on the entries of $\textbf{Z}$, which I've seen people call "sparse variable PCA"", but that's far less popular and to be honest I haven't thought about it that much. 
A: Whether sparse PCA is easier to interpret than standard PCA or not, depends on the dataset you are investigating. Here is how I think about it: sometimes one is more interested in the PCA projections (low dimensional representation of the data), and sometimes -- in the principal axes; it is only in the latter case that sparse PCA can have any benefits for the interpretation. Let me give a couple of examples.
I am e.g. working with neural data (simultaneous recordings of many neurons) and am applying PCA and/or related dimensionality reduction techniques to get a low-dimensional representation of neural population activity. I might have 1000 neurons (i.e. my data live in 1000-dimensional space) and want to project it on the three leading principal axes. What these axes are, is totally irrelevant for me, and I have no intention of "interpreting" these axes in any way. What I am interested, is the 3D projection (as the activity depends on time, I get a trajectory in this 3D space). So I am fine if each axis has all 1000 non-zero coefficients.
On the other hand, somebody might be working with more "tangible" data, where individual dimensions have obvious meaning (unlike individual neurons above). E.g. a dataset of various cars, where dimensions are anything from weight to price. In this case one might actually be interested in the leading principal axes themselves, because one might want to say something: look, the 1st principal axis corresponds to the "fanciness" of the car (I am totally making this up now). If the projection is sparse, such interpretations would generally be easier to give, because many variables will have $0$ coefficients and so are obviously irrelevant for this particular axis. In the case of standard PCA, one usually gets non-zero coefficients for all variables.
You can find more examples and some discussion of the latter case in the 2006 Sparse PCA paper by Zou et al. The difference between the former and the latter case, however, I did not see explicitly discussed anywhere (even though it probably was).
A: 
So you can eliminate the last few principal components, as they will not cause a lot of loss of data, and you can compress the data. Right?

yes, you're right. And if there are $N$ variables $V_1, V_2, \cdots , V_N$, you then have $N$ Principal Component $PC_1, PC_2, \cdots , PC_N$, and every variable $V_i$ has an information (a contribution) in every PC $PC_i$.
In Sparse PCA there are $PC_i$ without information of some variables $V_j, V_l, \cdots$, the variables with coefficient zero.
Then, if in one plane $(PC_i, PC_{j})$, there are fewer variables than expected ($N$), it's easier to clear the linear relations between them in this plane.   
A: Like all good things, it depends.

After applying PCA, you can again represent it in the same dimensional space, but, this time, the first principal component will contain the most variance, the second will contain the second most variance direction and so on. So you can eliminate the last few principal components, as they will not cause a lot of loss of data, and you can compress the data. Right?

Yes. Here's the thing: the same logic applies for the parameters of each principal components themselves. Just as many of the principal components can be eliminated without losing very much information, you can eliminate many parts of each principal component without losing very much information. Why should we be efficient in one dimension but not another? Eliminate all of the unnecessary parameters.
If you think about the total number of parameters needed to describe your data, sparse PCA is very often just more efficient PCA. Modern sparse PCA methods are a big reason modern video and image compression is so good and doesn't look like sh-t. Sparse PCA is also easier to interpret; each principal component can actually represent a descriptive factor of some kind (for example: personality traits in psychological studies). And finally, sparse PCA (like sparse regression) is more likely to generalize better out of sample. If I have some new data generated from the same source, sparse principal components are more likely to summarize the new data better (less is more when it comes to generalizing).
Of course, that depends on your measure of efficiency. If you're using PCA to visualize multi-dimensional data in two or three dimensions, sparse PCA would be less efficient (each principal component would describe less variance than simple PCA).
Tl;dr simple PCA = handful of complicated principal components, sparse PCA = slightly more simple principal components and less parameters in total.
