2
$\begingroup$

In "A new coefficient of correlation" (Chatterjee, 2019), Chatterjee defined an asymptotically unbiased estimator for the following index, which is zero for independent random variables $X,Y$ and one if $Y$ is a deterministic function of $X$: $$\xi(X,Y) = \frac{\int \mbox{Var}\left(E(1_{\\{Y\geq t\\}}|X)\right) d\mu(t) }{\int \mbox{Var}(1_{\\{Y\geq t\\}}) d\mu(t)}$$

where $\mu$ is the (unconditional) probability distribution of $Y$. The paper does not say so, but I guess that the variance in the numerator is taken with respect to $X$, not with respect to $t$, because otherwise the outer integral with respect to $t$ would make no sense.

As I am not very familiar with this notation, I would first like to ask whether my following understanding is correct:

  1. As $1_{\\{Y\geq t\\}}$ can only be one or zero, its expectaion value under the condition of $X$ is simply the conditional probability $P(Y\geq t|X)$.
  2. The variance in the denominator is the variance of a Bernoulli variable and thus $pq$, or $P(Y\geq t)(1-P(Y\geq t))$.
  3. The outer integrals are irrelevant for the property that $\xi$ is zero for independence of $X$ and $Y$ and one for strict dependency. They are just applied to make the value independent of $t$.

If my above interpretations are correct, the index $\xi$ can equivalently be written in terms of the unconditional distributions $\lambda$ for $X$ and $\mu$ for $Y$ as $$\xi(X,Y) = \frac{\int\int \Big( P(Y\geq t|X=x)-m(t)\Big)^2 d\lambda(x) d\mu(t)}{\int P(Y\geq t) \Big(1-P(Y\geq t)\Big) d\mu(t)} \quad\mbox{with}\quad m(t)=\int P(Y\geq t|X=x) d\lambda(x) = P(Y\geq t)$$

My second question is, what is the value of $\xi$ for the very simple model $Y=f(X) + \varepsilon$ where $X$ is uniformly or normally distributed, $f$ is a deterministic function, and $\varepsilon$ is a normally distributed noise with zero mean and variance $\sigma^2$. A simple Monte Carlo estimate can be achieved by simulating data and use Chatterjee's estimator, but I wonder whether the integrals can be solved in closed form in this simple situation.

$\endgroup$
4
  • $\begingroup$ I'm also lost with this notation. It's a shame that no one answered this. $\endgroup$
    – Juan
    Nov 30, 2023 at 20:56
  • $\begingroup$ @Juan In the meantime, I could verify that my interpretation is correct by computing the integrals numerically for a variety of models and comparing this true $\xi$ with the value of xicor() for a large data set generated according to the respective model. I am currently doing a number of tests with xicor for these models and can keep you informed about the results, if you are interested. $\endgroup$
    – cdalitz
    Dec 1, 2023 at 14:40
  • $\begingroup$ Oh! Well I think this us way above my level of understanding. In any case, it's always nice to share your own research, if you're feeling like it. $\endgroup$
    – Juan
    Dec 1, 2023 at 16:51
  • $\begingroup$ @Juan Meanhwile, we have written down our results. Please have a look at section 2 of this paper for a detailed explanation of writing Cahtterjee's $\xi$ in terms of conditional probabilities and how it is related to a similar quantity be Dette et al: arxiv.org/abs/2312.15496 $\endgroup$
    – cdalitz
    Dec 27, 2023 at 13:22

1 Answer 1

1
$\begingroup$

In order to mark this question as answered: Yes, the formulation with conditional probabilities is equivalent to Chatterjee's original formulation.

Interestingly, for continuous $Y$, some of the integrals can be readily computed because of $d\mu(t)=-P'(Y\geq t)\,dt$ where the prime denotes derivation with respect to $t$. This yields \begin{align} \int P(Y\!\geq\! t)^2 d\mu(t) &= -\frac{1}{3}P(Y\!\geq\! t)^3\Big|_{-\infty}^{\infty} = \frac{1}{3} \\ \int P(Y\!\geq\! t) d\mu(t) &= -\frac{1}{2}P(Y\!\geq\! t)^2\Big|_{-\infty}^{\infty} = \frac{1}{2} \end{align} which results in $$\xi(X,Y) = 6\int \!\!\!\int\! P(Y\!\geq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) - 2$$ Moreover, for continuous $Y$, the double integral can also be rewritten as \begin{align} \int \!\!\!\int\! & P(Y\!\geq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \\ &= \int \!\!\!\int\! \Big(1-P(Y\!\leq\! t|X\!=\!x)\Big)^2 d\lambda(x)\, d\mu(t) \\ &= 1 - 2\int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x) d\lambda(x)\, d\mu(t) \\ &\quad\quad + \int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \\ &= 1 - 2 \int P(Y\!\leq\! t)\, d\mu(t) + \int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \\ &= \int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) \end{align} which results in the following expression for $\xi$ that was already proposed in 2010 by Dette, Sieburg & Stoimenov as a measure for correlation between continuous random variables: $$\xi(X,Y) = 6\int \!\!\!\int\! P(Y\!\leq\! t|X\!=\!x)^2 d\lambda(x)\, d\mu(t) - 2$$ R code for computing $\xi$ for some models of $X$ and $Y$ can be found in the appendix of https://arxiv.org/abs/2312.15496.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.