Covariance between two random variables defines a measure of how closely are they linearly related to each other. But what if the joint distribution is circlular? Surely there is structure in the distribution. How is this structure extracted?
By "circular" I understand that the distribution is concentrated on a circular region, as in this contour plot of a pdf.
If such a structure exists, even partially, a natural way to identify and measure it is to average the distribution circularly around its center. (Intuitively, this means that for each possible radius $r$ we should spread the probability of being at distance $r$ from the center equally around in all directions.) Denoting the variables as $(X,Y)$, the center must be located at the point of first moments $(\mu_X, \mu_Y)$. To do the averaging it is convenient to define the radial distribution function
$$F(\rho) = \Pr[(X-\mu_X)^2 + (Y-\mu_Y)^2 \le \rho^2], \rho \ge 0;$$ $$F(\rho) = 0, \rho \lt 0.$$
This captures the total probability of lying between distance $0$ and $\rho$ of the center. To spread it out in all directions, let $R$ be a random variable with cdf $F$ and $\Theta$ be a uniform random variable on $[0, 2\pi]$ independent of $R$. The bivariate random variable $(\Xi, H) = (R\cos(\Theta) + \mu_X, R\sin(\Theta)+\mu_Y)$ is the circular average of $(X,Y)$. (This does the job our intuition demands of a "circular average" because (a) it has the correct radial distribution, namely $F$, by construction, and (b) all directions from the center ($\Theta$) are equally probable.)
At this point you have many choices: all that remains is to compare the distribution of $(X,Y)$ to that of $(\Xi, H)$. Possibilities include an $L^p$ distance and the Kullback-Leibler divergence (along with myriad related distance measures: symmetrized divergence, Hellinger distance, mutual information, etc.). The comparison suggests $(X,Y)$ may have a circular structure when it is "close" to $(\Xi, H)$. In this case the structure can be "extracted" from properties of $F$. For instance, a measure of central location of $F$, such as its mean or median, identifies the "radius" of the distribution of $(X,Y)$, and the standard deviation (or other measure of scale) of $F$ expresses how "spread out" $(X,Y)$ are in the radial directions about their central location $(\mu_X, \mu_Y)$.
When sampling from a distribution, with data $(x_i,y_i), 1 \le i \le n$, a reasonable test of circularity is to estimate the central location as usual (with means or medians) and thence convert each value $(x_i,y_i)$ into polar coordinates $(r_i, \theta_i)$ relative to that estimated center. Compare the standard deviation (or IQR) of the radii to their mean (or median). For non-circular distributions the ratio will be large; for circular distributions it should be relatively small. (If you have a specific model in mind for the underlying distribution, you can work out the sampling distribution of the radial statistic and construct a significance test with it.) Separately, test the angular coordinate for uniformity in the interval $[0, 2\pi)$. It will be approximately uniform for circular distributions (and for some other distributions, too); non-uniformity indicates a departure from circularity.
Mutual information has properties somewhat analogous to covariance. Covariance is a number which is 0 for independent variables and nonzero for variables which are linearly dependent. In particular, if two variables are the same, then the covariance is equal to variance (which is usually a positive number). One issue with covariance is that it may be zero even if two variables are not independent, provided the dependence is nonlinear.
Mutual information (MI) is a non-negative number. It is zero if and only if the two variables are statistically independent. This property is more general than that of covariance and covers any dependencies, including nonlinear ones.
If the two variable are the same, MI is equal to the variable's entropy (again, usually a positive number). If the variables are different and not deterministically related, then MI is smaller than the entropy. In this sense, MI of two variables goes between 0 and H (the entropy), with 0 only if independent and H only if deterministically dependent.
One difference from covariance is that the "sign" of dependency is ignored. E.g. $Cov(X, -X) = -Cov(X, X) = -Var(X)$, but $MI(X, -X) = MI(X, X) = H(X)$.
Please have a look at the following article from science - it addresses your point exactly:
From the abstract:
You find supplemental material here: http://www.sciencemag.org/content/suppl/2011/12/14/334.6062.1518.DC1
The authors even provide a free tool incorporating the novel method which can be used with R and Python: http://www.exploredata.net/