In the context of this paper, it appears as if they are using to ‘second order dependencies’ to refer to cases where $X$ and $Y$ are orthogonal to each other, and higher order dependencies when $X$ and $Y$ are not. Finding orthogonal axes is the basis for the principal component analysis, because you are trying to find what orthogonal axes exist that can explain the maximum amount of variation. Their point is that for some more complicated data sets, looking for orthogonal axes does not really make sense because it may systematically explain too little of the information. I think it’s easiest to explain this with a (terrible) MS Paint picture:
Taking their Figure 6 and butchering it, their point is that if you have an orthogonal $X$ and $Y$ then there are only 4 positions on the ferris wheel that this system can explain. There are many other positions in between where you have $Y$ and some combo of $Y$ and $X$ (or $X$ and some combo of $Y$ and $X$) required to explain it, which is sort of like needing $Y^2$ and $X$ to explain the information (aka, a higher-order dependency). In this case if you were aware that your data described a circular path, the rational solution would be to use $\Theta$ to describe it and avoid PCA (which is their first suggested solution - to use $a$ $priori$ information). But you can't always predict higher order relationships.