I am looking for distances between two random variables $X$ and $Y$, or practical estimates for measuring the distance between the i.i.d. observations $(X^1, \ldots, X^T)$ and $(Y^1,\ldots,Y^T)$. I am aware of the divergences or statistical distances, but they focus on quantifying the dissimilarity in distribution which is sufficient when $X$ and $Y$ are independent, but fail to measure how "correlated" they are otherwise. Any information is welcomed!
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ This needs to be tightened up a bit. What property of two random variables would result in zero distance? $\endgroup$– user75138Commented Sep 13, 2015 at 2:18
-
$\begingroup$ A perfect dependence between the two random variables AND the same distribution would result in zero distance. Would you suggest any other property? I think that a reasonable distance should take into account dependence and distribution since it amounts for the whole distribution according to Sklar's theorem in copula theory. $\endgroup$– micCommented Sep 13, 2015 at 19:22
-
$\begingroup$ So if we have such a distance $d(\mathbf{X,Y})$ then $d=0$ iff $X,Y$ come from the same distribution and are perfectly (linearly?) correlated? $\endgroup$– user75138Commented Sep 13, 2015 at 20:01
-
$\begingroup$ Not necessarily linearly correlated, it could be "linearly" correlated up to some monotonous transforms or even a broader notion of "correlation". $\endgroup$– micCommented Sep 13, 2015 at 22:20
-
$\begingroup$ I don't understand your criticism of divergence measures... Why doesn't mutual information provide you 'non linear' correlation $\endgroup$– seanv507Commented Sep 13, 2015 at 23:27
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
2
Here's a measure that seems to accord with your requirements for the case of monotonic relationships between $X$ and $Y$:
Let $X,Y$ be your sample vectors. Let $S(X,Y)$ be the Spearman Rank Correlation between these two vectors and let $KS(X,Y)$ be the Kolmorogov-Smirnov Statistic between the EDCF of $X$ and ECDF of $Y$.
We can construct the quantity $D(X,Y)=||S(X,Y)|-1|+KS(X,Y)$. Lets analyze the cases:
- If $Y=f(X)$ where $f()$ is monotonic, then $KS(X,Y)=0 and|S(X,Y)|=1$, so the $D(X,Y)=0$.
- If $X,Y$ both come from the same distribution but are uncorrelated, then $S(X,Y)\approx 0$ and $KS(X,Y)\approx 0$ for large enough samples, so $D(X,Y)\approx 1$
- If $X,Y$ are neither correlated nor from the same distribution then $D(X,Y) \in [0,2]$.
So, this is more of a "coefficient" or index than a distance, but maybe itll work for you.
-
$\begingroup$ Thanks Bey. Actually, your idea is quite close to this arxiv paper. But I am not entirely satisfied with that. 1) You cannot control easily the weighting between dependence and distribution. 2) Why a convex combination of the two parts? One could have used generalized-means... 3) despite the two parts live in $[0,1]$, one does not control how they vary in these intervals and these two distances might be rather inhomogeneous! $\endgroup$– micCommented Sep 14, 2015 at 7:15
-
$\begingroup$ @mic your need for so much control undermines your goal of having an objective metric of closeness. For example, there are weightings that will contradict your goal of having zero distance for perfect monotonic correlation. You really need to think about the specifics of what you need...what decision(s) will this metric be supporting? What are the risks of wrong decisions? There is quite a bit of literature on Statistical Decision Theory that can help turn a vague, subjective requirement into something precise. $\endgroup$– user75138Commented Sep 14, 2015 at 12:50