# Measuring non-linear dependence

Covariance between two random variables defines a measure of how closely are they linearly related to each other. But what if the joint distribution is circlular? Surely there is structure in the distribution. How is this structure extracted?

-

By "circular" I understand that the distribution is concentrated on a circular region, as in this contour plot of a pdf.

If such a structure exists, even partially, a natural way to identify and measure it is to average the distribution circularly around its center. (Intuitively, this means that for each possible radius $r$ we should spread the probability of being at distance $r$ from the center equally around in all directions.) Denoting the variables as $(X,Y)$, the center must be located at the point of first moments $(\mu_X, \mu_Y)$. To do the averaging it is convenient to define the radial distribution function

$$F(\rho) = \Pr[(X-\mu_X)^2 + (Y-\mu_Y)^2 \le \rho^2], \rho \ge 0;$$ $$F(\rho) = 0, \rho \lt 0.$$

This captures the total probability of lying between distance $0$ and $\rho$ of the center. To spread it out in all directions, let $R$ be a random variable with cdf $F$ and $\Theta$ be a uniform random variable on $[0, 2\pi]$ independent of $R$. The bivariate random variable $(\Xi, H) = (R\cos(\Theta) + \mu_X, R\sin(\Theta)+\mu_Y)$ is the circular average of $(X,Y)$. (This does the job our intuition demands of a "circular average" because (a) it has the correct radial distribution, namely $F$, by construction, and (b) all directions from the center ($\Theta$) are equally probable.)

At this point you have many choices: all that remains is to compare the distribution of $(X,Y)$ to that of $(\Xi, H)$. Possibilities include an $L^p$ distance and the Kullback-Leibler divergence (along with myriad related distance measures: symmetrized divergence, Hellinger distance, mutual information, etc.). The comparison suggests $(X,Y)$ may have a circular structure when it is "close" to $(\Xi, H)$. In this case the structure can be "extracted" from properties of $F$. For instance, a measure of central location of $F$, such as its mean or median, identifies the "radius" of the distribution of $(X,Y)$, and the standard deviation (or other measure of scale) of $F$ expresses how "spread out" $(X,Y)$ are in the radial directions about their central location $(\mu_X, \mu_Y)$.

When sampling from a distribution, with data $(x_i,y_i), 1 \le i \le n$, a reasonable test of circularity is to estimate the central location as usual (with means or medians) and thence convert each value $(x_i,y_i)$ into polar coordinates $(r_i, \theta_i)$ relative to that estimated center. Compare the standard deviation (or IQR) of the radii to their mean (or median). For non-circular distributions the ratio will be large; for circular distributions it should be relatively small. (If you have a specific model in mind for the underlying distribution, you can work out the sampling distribution of the radial statistic and construct a significance test with it.) Separately, test the angular coordinate for uniformity in the interval $[0, 2\pi)$. It will be approximately uniform for circular distributions (and for some other distributions, too); non-uniformity indicates a departure from circularity.

-
Thank you! Although not entirely clear, this does give me some idea. Could you please recommend some reading where these kind of distributions are tackled? I have been only exposed to Gaussians and the other standard distributions. Another question, does this have something to do with the radial distribution functions of the atoms etc? – Infinity Apr 10 '11 at 23:24
@Infinity Let me know what part is not clear so I can try to fix it up. I don't know where such distributions are discussed, but related analysis can be found in the literature on "circular distributions." The underlying mathematical ideas are indeed somewhat tenuously related to atomic orbital theory. The relevant concepts include separability of the Schrodinger Equation in spherical coordinates, constructing Haar measure of a compact Lie group by averaging, and comparing orbitals by means of overlap integrals. – whuber Apr 11 '11 at 1:43
Thanks. I am very new to probability and stats so it was probably because of that. I don't really understand what you mean by "average the distribution circularly around its center", I think it means to average out all the circles so that there is only one circle left with center at $(\mu_X, \mu_Y)$ and radius $\rho$ kinda like a linear regression line fit. Is that correct? – Infinity Apr 11 '11 at 8:42
The other doubt I have is that the distribution function $F(\rho)$ seems to describe a disc but the figure (and what I had in mind) is a ring. The random variable $(\Xi, H)$ describes the average circle in polar form. I am sorry I do not clearly get what happens next. I understand we compare the two distributions using some distance metric, but why is the $(\Xi, H)$ special and how it helps I am unable to reason. I am sorry if the questions seem too stupid. – Infinity Apr 11 '11 at 8:55
@Infinity I added some clarifying remarks. You don't average out circles; rather, you average out (or "smear") all the probability across each circle so that no matter what you started with, it ends up looking like my picture (with circular contours). If the original distribution was truly circular, this averaging doesn't change it. Thus, comparing the distribution to its averaged version tells you how far it is from being circular in the first place. – whuber Apr 11 '11 at 14:55

Detecting Novel Associations in Large Data Sets by David N. Reshef et al.

From the abstract:

Identifying interesting relationships between pairs of variables in large data sets is increasingly important. Here, we present a measure of dependence for two-variable relationships: the maximal information coefficient (MIC). MIC captures a wide range of associations both functional and not, and for functional relationships provides a score that roughly equals the coefficient of determination (R^2) of the data relative to the regression function. MIC belongs to a larger class of maximal information-based nonparametric exploration (MINE) statistics for identifying and classifying relationships. We apply MIC and MINE to data sets in global health, gene expression, major-league baseball, and the human gut microbiota and identify known and novel relationships.

You find supplemental material here: http://www.sciencemag.org/content/suppl/2011/12/14/334.6062.1518.DC1

The authors even provide a free tool incorporating the novel method which can be used with R and Python: http://www.exploredata.net/

-

Mutual information has properties somewhat analogous to covariance. Covariance is a number which is 0 for independent variables and nonzero for variables which are linearly dependent. In particular, if two variables are the same, then the covariance is equal to variance (which is usually a positive number). One issue with covariance is that it may be zero even if two variables are not independent, provided the dependence is nonlinear.

Mutual information (MI) is a non-negative number. It is zero if and only if the two variables are statistically independent. This property is more general than that of covariance and covers any dependencies, including nonlinear ones.

If the two variable are the same, MI is equal to the variable's entropy (again, usually a positive number). If the variables are different and not deterministically related, then MI is smaller than the entropy. In this sense, MI of two variables goes between 0 and H (the entropy), with 0 only if independent and H only if deterministically dependent.

One difference from covariance is that the "sign" of dependency is ignored. E.g. $Cov(X, -X) = -Cov(X, X) = -Var(X)$, but $MI(X, -X) = MI(X, X) = H(X)$.

-
Could you expand on how this concept provides an answer to the question? – onestop Apr 11 '11 at 7:02