Chatterjee's Correlation: Understanding & Implementation

Question

As a lay person I'm having trouble understanding how Chatterjee's formula as defined produces a correlation between two time series when it only references one of them. Pearson's/Spearman's correlations both reference the separate series $(X,Y)$, but this one doesn't.

Let $r_i$ be the rank of $Y_{(i)}$, that is, the number of $j$ such that $Y_{(j)} <= Y_{(i)}$

...additionally define $l_i$ to be the number of $j$ such that $Y_{(j)} >= Y_{(i)}$

There are only 3 unique terms in the formula: $r_i$, $l_i$, and $n$ (which seems to be just the # of elements). So based on what the formula itself shows, X never seems to be used.

$$ E(X,Y) = 1 - \frac{n \sum^{n-1}_{i=1}{|r_{i+1} - r_i|}}{2 \sum{^n_{i=1}l_i(n-l_i)}} $$

Answer 1

The association between $X$ and $Y$ is captured by sorting the ranks of $Y$, $r_i$, using the ranks of $X$. Therefore when using $r_i$ we are using information from both $X$ and $Y$ despite $r_i$ being on face value, only dependent on $Y$. Here is a short example reproducing a result from the linked paper using Eq. (1.1):

set.seed(123)
N = 100
x = runif(N, -1, 1)
y = x^2 

DD = data.frame(x= x, y=y)
DD$r = rank(DD$y)
DD = DD[order(DD$x), ] # re-arrange rank(Y) based on X

1 - 3*sum(abs(diff(DD$r))) / (N^2-1) 
    # ~94.089 << Matches the 94.1% in Fig. 2(d)