Deciphering and computing the limiting value of Chatterjee's xicor?

Question

In "A new coefficient of correlation" (Chatterjee, 2019), Chatterjee defined an asymptotically unbiased estimator for the following index, which is zero for independent random variables $X,Y$ and one if $Y$ is a deterministic function of $X$: $$\xi(X,Y) = \frac{\int \mbox{Var}\left(E(1_{\\{Y\geq t\\}}|X)\right) d\mu(t) }{\int \mbox{Var}(1_{\\{Y\geq t\\}}) d\mu(t)}$$

where $\mu$ is the (unconditional) probability distribution of $Y$. The paper does not say so, but I guess that the variance in the numerator is taken with respect to $X$, not with respect to $t$, because otherwise the outer integral with respect to $t$ would make no sense.

As I am not very familiar with this notation, I would first like to ask whether my following understanding is correct:

1. As $1_{\\{Y\geq t\\}}$ can only be one or zero, its expectaion value under the condition of $X$ is simply the conditional probability $P(Y\geq t|X)$.
2. The variance in the denominator is the variance of a Bernoulli variable and thus $pq$, or $P(Y\geq t)(1-P(Y\geq t))$.
3. The outer integrals are irrelevant for the property that $\xi$ is zero for independence of $X$ and $Y$ and one for strict dependency. They are just applied to make the value independent of $t$.

If my above interpretations are correct, the index $\xi$ can equivalently be written in terms of the unconditional distributions $\lambda$ for $X$ and $\mu$ for $Y$ as $$\xi(X,Y) = \frac{\int\int \Big( P(Y\geq t|X=x)-m(t)\Big)^2 d\lambda(x) d\mu(t)}{\int P(Y\geq t) \Big(1-P(Y\geq t)\Big) d\mu(t)} \quad\mbox{with}\quad m(t)=\int P(Y\geq t|X=x) d\lambda(x) = P(Y\geq t)$$

My second question is, what is the value of $\xi$ for the very simple model $Y=f(X) + \varepsilon$ where $X$ is uniformly or normally distributed, $f$ is a deterministic function, and $\varepsilon$ is a normally distributed noise with zero mean and variance $\sigma^2$. A simple Monte Carlo estimate can be achieved by simulating data and use Chatterjee's estimator, but I wonder whether the integrals can be solved in closed form in this simple situation.

Answer 1

In order to mark this question as answered: Yes, the formulation with conditional probabilities is equivalent to Chatterjee's original formulation.

Interestingly, for continuous $Y$, some of the integrals can be readily computed because of $d\mu(t)=-P'(Y\geq t)\,dt$ where the prime denotes derivation with respect to $t$. This yields \begin{align} \int P(Y\!\geq\! t)^2 d\mu(t) &= -\frac{1}{3}P(Y\!\geq\! t)^3\Big|_{-\infty}^{\infty} = \frac{1}{3} \\ \int P(Y\!\geq\! t) d\mu(t) &= -\frac{1}{2}P(Y\!\geq\! t)^2\Big|_{-\infty}^{\infty} = \frac{1}{2} \end{align} which results in $$\xi(X,Y) = 6\int \!\!\!\int\! P(Y\!\geq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) - 2$$ Moreover, for continuous $Y$, the double integral can also be rewritten as \begin{align} \int \!\!\!\int\! & P(Y\!\geq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \\ &= \int \!\!\!\int\! \Big(1-P(Y\!\leq\! t|X\!=\!x)\Big)^2 d\lambda(x)\, d\mu(t) \\ &= 1 - 2\int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x) d\lambda(x)\, d\mu(t) \\ &\quad\quad + \int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \\ &= 1 - 2 \int P(Y\!\leq\! t)\, d\mu(t) + \int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \\ &= \int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \end{align} which results in the following expression for $\xi$ that was already proposed in 2010 by Dette, Sieburg & Stoimenov as a measure for correlation between continuous random variables: $$\xi(X,Y) = 6\int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) - 2$$ R code for computing $\xi$ for some models of $X$ and $Y$ can be found in the appendix of https://arxiv.org/abs/2312.15496.