# What are “second-order dependencies” and “higher order dependencies” in the data?

I am reading A Tutorial on Principal Component Analysis by Shlens, 2014, and it mentions these two notions: "second-order dependencies" and "higher order dependencies". I could not find any clear explanation of them. What do they mean?

The goal of the analysis is to decorrelate the data, or said in other terms, the goal is to remove second-order dependencies in the data. In the data sets of Figure 6, higher order dependencies exist between the variables. Therefore, removing second-order dependencies is insufficient at revealing all structure in the data.

PCA is based on variances and covariances, $\mathrm E[x_i x_j]$ (assuming mean-free variables). These are measures of second-order dependencies because the data enter in the form of terms of order 2. After PCA, the principal components have 0 covariance between them, so second-order dependencies have been removed. However, it is still possible that higher-order dependencies exist, e.g. that $\mathrm E[x_i x_j x_k] \neq 0$ for some $i$, $j$, and $k$. By removing second-order dependencies by applying a linear transform, PCA in a way "reveals" second-order dependencies in the form of that transform, but it does not "reveal" higher-order dependencies.
In the context of this paper, it appears as if they are using to ‘second order dependencies’ to refer to cases where $X$ and $Y$ are orthogonal to each other, and higher order dependencies when $X$ and $Y$ are not. Finding orthogonal axes is the basis for the principal component analysis, because you are trying to find what orthogonal axes exist that can explain the maximum amount of variation. Their point is that for some more complicated data sets, looking for orthogonal axes does not really make sense because it may systematically explain too little of the information. I think it’s easiest to explain this with a (terrible) MS Paint picture:
Taking their Figure 6 and butchering it, their point is that if you have an orthogonal $X$ and $Y$ then there are only 4 positions on the ferris wheel that this system can explain. There are many other positions in between where you have $Y$ and some combo of $Y$ and $X$ (or $X$ and some combo of $Y$ and $X$) required to explain it, which is sort of like needing $Y^2$ and $X$ to explain the information (aka, a higher-order dependency). In this case if you were aware that your data described a circular path, the rational solution would be to use $\Theta$ to describe it and avoid PCA (which is their first suggested solution - to use $a$ $priori$ information). But you can't always predict higher order relationships.