48
$\begingroup$

Suppose I have a sample $(X_n,Y_n), n=1..N$ from the joint distribution of $X$ and $Y$. How do I test the hypothesis that $X$ and $Y$ are independent?

No assumption is made on the joint or marginal distribution laws of $X$ and $Y$ (least of all joint normality, since in that case independence is identical to correlation being $0$).

No assumption is made on the nature of a possible relationship between $X$ and $Y$; it may be non-linear, so the variables are uncorrelated ($r=0$) but highly co-dependent ($I=H$).

I can see two approaches:

  1. Bin both variables and use Fisher's exact test or G-test.

    • Pro: use well-established statistical tests
    • Con: depends on binning
  2. Estimate the dependency of $X$ and $Y$: $\frac{I(X;Y)}{H(X,Y)}$ (this is $0$ for independent $X$ and $Y$ and $1$ when they completely determine each other).

    • Pro: produces a number with a clear theoretical meaning
    • Con: depends on the approximate entropy computation (i.e., binning again)

Do these approaches make sense?

What other methods people use?

$\endgroup$
  • 3
    $\begingroup$ Look into distance correlation. $\endgroup$ – Ray Koopman Oct 24 '13 at 18:18
  • $\begingroup$ @RayKoopman: thanks, I am reading Measuring and Testing Dependence by Correlation of Distances now! $\endgroup$ – sds Oct 24 '13 at 18:45
  • 2
    $\begingroup$ the dependency $I\left(X;Y\right)/H\left(X;Y\right)$ does not make sense when talking about continuous variables. Continuous variables have infinite entropy. Here, you can't substitute $H$ for the differential entropy, because the differential entropy is not comparable to mutual information. While mutual information has an "absolute" meaning, the differential entropy could be positive, zero, or even negative, depending on the units you use to measure the variables $X$ and $Y$. $\endgroup$ – fonini Jan 11 '17 at 2:22
  • $\begingroup$ @fonini: of course, I was talking about binned variables. Thanks for your comment though. $\endgroup$ – sds Jan 11 '17 at 4:46
27
$\begingroup$

This is a very hard problem in general, though your variables are apparently only 1d so that helps. Of course, the first step (when possible) should be to plot the data and see if anything pops out at you; you're in 2d so this should be easy.

Here are a few approaches that work in $\mathbb{R}^n$ or even more general settings:

$\endgroup$
  • $\begingroup$ Can you briefly mention how these approaches compare to Distance Correlation? I'm using DC to sift through large datasets (well, large for me), so I'm interested in any comments you might have. Thanks! $\endgroup$ – pteetor Oct 25 '13 at 14:53
  • 1
    $\begingroup$ @pteetor That's interesting, I hadn't run across distance correlation before. Computationally, it seems more expensive than the entropy estimation approach for large sample sizes because you need the full distance matrices (where for the entropy estimators you can use indices to get only the first k neighbors). No idea how it compares in terms of statistical power / etc. $\endgroup$ – Dougal Oct 29 '13 at 19:56
  • 4
    $\begingroup$ For later readers: The 2013 paper Equivalence of distance-based and RKHS-based statistics in hypothesis testing by Sejdinovic et al. shows that distance correlation and other energy distances are particular instances of MMD, the underlying measure behind HSIC, and discusses the relationship in terms of test power and so on. $\endgroup$ – Dougal Feb 1 '16 at 16:18
19
$\begingroup$

Hoeffding developed a general nonparametric test for the independence of two continuous variables using joint ranks to test $H_{0}: H(x,y) = F(x)G(y)$. This 1948 test is implemented in the R Hmisc package's hoeffd function.

$\endgroup$
6
$\begingroup$

How about this paper:

http://arxiv.org/pdf/0803.4101.pdf

"Measuring and testing dependence by correlation of distances". Székely and Bakirov always have interesting stuff.

There is matlab code for the implementation:

http://www.mathworks.com/matlabcentral/fileexchange/39905-distance-correlation

If you find any other (simple to implement) test for independence let us know.

$\endgroup$
  • 2
    $\begingroup$ Welcome to the site, @JLp. We hope to build a permanent repository of high-quality statistical information in the form of questions & answers. As such, one thing we worry about is linkrot. With that in mind, would you mind giving a summary of what is in that paper / how it answers the questions, in case the link were to go dead. It will also help future readers of this thread decide whether they want to invest the time to read the paper. $\endgroup$ – gung - Reinstate Monica Oct 25 '13 at 15:35
  • $\begingroup$ @gung: this is the same as energy $\endgroup$ – sds Oct 27 '13 at 17:51
5
$\begingroup$

The link between Distance Covariance and kernel tests (based on the Hilbert-Schmidt independence criterion) is given in the paper:

Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K., Equivalence of distance-based and RKHS-based statistics in hypothesis testing, Annals of Statistics, 41 (5), pp.2263-2702, 2013

It's shown that distance covariance is a special case of the kernel statistic, for a particular family of kernels.

If you're intent on using mutual information, a test based on a binned estimate of the MI is:

Gretton, A. and Gyorfi, L., Consistent Nonparametric Tests of Independence, Journal of Machine Learning Research, 11 , pp.1391--1423, 2010.

If you're interested in getting the best test power, you're better off using the kernel tests, rather than binning and mutual information.

That said, given your variables are univariate, classical nonparametric independence tests like Hoeffding's are probably fine.

$\endgroup$
4
$\begingroup$

Rarely (never?) in statistics can you demonstrate that your sample statistic = a point value. You can test against point values and either exclude them or not exclude them. But the nature of statistics is that it is about examining variable data. Because there is always variance then there will necessarily be no way to know that something is exactly not related, normal, gaussian, etc. You can only know a range of values for it. You could know if a value is excluded from the range of plausible values. For example, it's easy to exclude no relationship and give range of values for how big the relationship is.

Therefore, trying to demonstrate no relationship, essentially the point value of relationship = 0 is not going to meet with success. If you have a range of measures of relationship that are acceptable as approximately 0. Then it would be possible to devise a test.

Assuming that you can accept that limitation it would be helpful to people trying to assist you to provide a scatterplot with a lowess curve. Since you're looking for R solutions try:

scatter.smooth(x, y)

Based on the limited information you've given so far I think a generalized additive model might be the best thing for testing non-independence. If you plot that with CI's around the predicted values you may be able to make statements about a belief of independence. Check out gam in the mgcv package. The help is quite good and there is assistance here regarding the CI.

$\endgroup$
2
$\begingroup$

It may be interesting ...

Garcia, J. E.; Gonzalez-Lopez, V. A. (2014) Independence tests for continuous random variables based on the longest increasing subsequence. Journal of Multivariate Analysis, v. 127 p. 126-146.

http://www.sciencedirect.com/science/article/pii/S0047259X14000335

$\endgroup$
-1
$\begingroup$

If you use R, cor.test function in stats package (default in R) can do that:

Test for Association/Correlation Between Paired Samples. Test for association between paired samples, using one of Pearson's product moment correlation coefficient, Kendall's tau or Spearman's rho.

cor.test(x, y,method="spearman")
$\endgroup$
  • 1
    $\begingroup$ This misses the non-linear relationships which are explicitly the topic of the question. $\endgroup$ – sds Nov 10 at 4:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.