# Detecting non-linear relationships between distributions?

X,Y are r.v's exactly related by some unknown non-linear relationship.

Does there exist a neat analog of Correlation, that gives some information about a this relationship?

• @whuber Okay, maybe just an example of quadratic relationship. I want to get a feel for the methods people use in this area. – Cris Stringfellow Mar 14 '13 at 4:23
• Cris, are you now saying that $X$ and $Y$ are known to have a quadratic relationship (perhaps up to some random iid error) or are you saying that this is the relationship but it is unknown? If you mean the latter, then your edit hasn't changed anything. If you mean the former, then the answer is "regression," which is abundantly answered in hundreds of questions here. – whuber Mar 14 '13 at 4:27
• @whuber Okay so in the case of the former one can just try out different functions and use regression. Makes sense. One could simply try out a whole bunch of different polynomial, exponential, trigonometric models. There can't be that many templates, and regression can find the parameters. That answers that part. For the latter case where there is no 'closed form' relationship... I guess, it is data mining. – Cris Stringfellow Mar 14 '13 at 4:34
• It's not quite like that, Cris. In the former case all you try is a quadratic regression: there's no call for anything else. (There are infinitely many "templates," anyway, and an infinite subset of them will fit the data perfectly, so the process of trying out lots of things in an aimless or automatic way is a sterile, ultimately misleading endeavor.) In the latter case you're essentially discussing a theory of everything, which includes data mining, cross-validation, model selection, machine learning, exploratory data analysis, and much more. – whuber Mar 14 '13 at 5:05
• @whuber: Isn't Symbolic Regression doing exactly what you describe? Using templates to find a best fit in an automatic way? e.g. Something like the Eureqa Package from Cornell creativemachines.cornell.edu/eureqa Or is that different? – curious_cat Mar 14 '13 at 6:48

Were you looking for something like $Distance Correlation$?

http://en.wikipedia.org/wiki/Distance_correlation

This will be non-zero for any sort of relationship between your $x$ and $y$. Therefore this can trap arbitrary non-linear relationships though interpreting the values is harder than interpreting correlation.

If this is what you need try the "energy" library in R.

set.seed(1234)
x<-rnorm(1000,0,1)
y<-x^2
cor(x,y)
 -0.03908369
library(energy)
dcor(x,y, R=500))
 0.5478997

#to get a p-value for the distance correlation:
dcov.test(x,y)

dCov test of independence

data:  index 1, replicates 500
nV^2 = 140.74, p-value = 0.001996
sample estimates:
dCov
0.3751596

• This sounds good. – Cris Stringfellow Mar 14 '13 at 10:23
• Curious cat I am guessing that the reason the distance covariance gives a score 50 times greater than the correlation is because one can say a normal distribution is 'somewhat close in shape to a parabola.', but that this relationship is not a linear one, though it can be somewhat detected by distance corelation. Does that interpretation work, intuitively? – Cris Stringfellow Mar 14 '13 at 10:56
• @Chris: Could be. I can't say. I don't know enough about the theory behind it. – curious_cat Mar 14 '13 at 19:01

Correlation using Spearmans (captures non-linear relationships)

corr = df.corr("spearman")


In the case of monotonic relationship, the variables tend to move in the same relative direction, but not necessarily at a constant rate. In a linear relationship, the variables move in the same direction at a constant rate. In some cases, variables increasing concurrently, but not at the same rate. This relationship is monotonic, but not linear.

The Pearson correlation coefficient for these data is 0.843, but the Spearman correlation is higher 0.948. 