65
$\begingroup$

Suppose I have a sample $(X_n,Y_n), n=1..N$ from the joint distribution of $X$ and $Y$. How do I test the hypothesis that $X$ and $Y$ are independent?

No assumption is made on the joint or marginal distribution laws of $X$ and $Y$ (least of all joint normality, since in that case independence is identical to correlation being $0$).

No assumption is made on the nature of a possible relationship between $X$ and $Y$; it may be non-linear, so the variables are uncorrelated ($r=0$) but highly co-dependent ($I=H$).

I can see two approaches:

  1. Bin both variables and use Fisher's exact test or G-test.

    • Pro: use well-established statistical tests
    • Con: depends on binning
  2. Estimate the dependency of $X$ and $Y$: $\frac{I(X;Y)}{H(X,Y)}$ (this is $0$ for independent $X$ and $Y$ and $1$ when they completely determine each other).

    • Pro: produces a number with a clear theoretical meaning
    • Con: depends on the approximate entropy computation (i.e., binning again)

Do these approaches make sense?

What other methods people use?

$\endgroup$
4
  • 4
    $\begingroup$ Look into distance correlation. $\endgroup$ Oct 24, 2013 at 18:18
  • 2
    $\begingroup$ @RayKoopman: thanks, I am reading Measuring and Testing Dependence by Correlation of Distances now! $\endgroup$
    – sds
    Oct 24, 2013 at 18:45
  • 2
    $\begingroup$ the dependency $I\left(X;Y\right)/H\left(X;Y\right)$ does not make sense when talking about continuous variables. Continuous variables have infinite entropy. Here, you can't substitute $H$ for the differential entropy, because the differential entropy is not comparable to mutual information. While mutual information has an "absolute" meaning, the differential entropy could be positive, zero, or even negative, depending on the units you use to measure the variables $X$ and $Y$. $\endgroup$
    – fonini
    Jan 11, 2017 at 2:22
  • $\begingroup$ @fonini: of course, I was talking about binned variables. Thanks for your comment though. $\endgroup$
    – sds
    Jan 11, 2017 at 4:46

7 Answers 7

39
$\begingroup$

(Answer partially updated in 2023.)

This is a very hard problem in general, though your variables are apparently only 1d so that helps. Of course, the first step (when possible) should be to plot the data and see if anything pops out at you; you're in 2d so this should be easy.

Here are a few approaches that work in $\mathbb{R}^d$ or even more general settings, to match the general title of the question.

One general category is, related to the suggestion here, to estimate the mutual information:

  • Estimate mutual information via entropies, as mentioned. In low dimensions with sufficient samples, histograms / KDE / nearest-neighbour estimators should work okay, but expect them to behave very poorly as the dimension increases. In particular, the following simple estimator has finite-sample bounds (compared to most approaches' asymptotic-only properties):

Sricharan, Raich, and Hero. Empirical estimation of entropy functionals with confidence. arXiv:1012.4188 [math.ST]

  • Similar direct estimators of mutual information, e.g. the following based on nearest neighbours:

Pál, Póczos, and Svepesári. Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs, NeurIPS 2010.

  • Variational estimators of mutual information, based on optimizing some function parameterized typically as a neural network; this is probably the "default" modern approach in high dimensions. The following paper gives a nice overview of the relationship between various estimators. Be aware, however, that these approaches are highly dependent on the neural network class and optimization scheme, and can have particularly surprising behaviour in their bias/variance tradeoffs.

Poole, Ozair, van den Oord, Alemi, and Tucker. On Variational Bounds of Mutual Information, ICML 2019.

There are also other approaches, based on measures other than the mutual information.

  • The Schweizer-Wolff approach is a classic one based on copula transformations, and so is invariant to monotone increasing transformations. I'm not very familiar with this one, but I think it's computationally simpler but also maybe less powerful than most of the other approaches here. (I vaguely expect it can be framed as a special case of some of the other approaches but haven't really thought about it.)

Schweizer and Wolff, On Nonparametric Measures of Dependence for Random Variables, Annals of Statistics 1981.

  • The Hilbert-Schmidt independence criterion (HSIC): a kernel (in the sense of RKHS, not KDE)-based approach, based on measuring the norm of $\operatorname{Cov}(\phi(X), \psi(Y))$ for kernel features $\phi$ and $\psi$. In fact, the HSIC with kernels defined by a deep network is related to one of the more common variational estimators, InfoNCE; see discussion here.

Gretton, Bousqet, Smola, and Schölkopf, Measuring Statistical Independence with Hilbert-Schmidt Norms, Algorithmic Learning Theory 2005.

  • Statisticians are probably more familiar with the distance covariance/correlation as mentioned here previously; this is in fact a special case of the HSIC with a particular choice of kernel, but that choice is maybe often a better kernel choice than the default Gaussian kernel typically used for HSIC.

Székely, Rizzo, and Bakirov, Measuring and testing dependence by correlation of distances, Annals of Statistics 2007.

$\endgroup$
3
  • 1
    $\begingroup$ Can you briefly mention how these approaches compare to Distance Correlation? I'm using DC to sift through large datasets (well, large for me), so I'm interested in any comments you might have. Thanks! $\endgroup$
    – pteetor
    Oct 25, 2013 at 14:53
  • 2
    $\begingroup$ @pteetor That's interesting, I hadn't run across distance correlation before. Computationally, it seems more expensive than the entropy estimation approach for large sample sizes because you need the full distance matrices (where for the entropy estimators you can use indices to get only the first k neighbors). No idea how it compares in terms of statistical power / etc. $\endgroup$
    – Danica
    Oct 29, 2013 at 19:56
  • 5
    $\begingroup$ For later readers: The 2013 paper Equivalence of distance-based and RKHS-based statistics in hypothesis testing by Sejdinovic et al. shows that distance correlation and other energy distances are particular instances of MMD, the underlying measure behind HSIC, and discusses the relationship in terms of test power and so on. $\endgroup$
    – Danica
    Feb 1, 2016 at 16:18
25
$\begingroup$

Hoeffding developed a general nonparametric test for the independence of two continuous variables using joint ranks to test $H_{0}: H(x,y) = F(x)G(y)$. This 1948 test is implemented in the R Hmisc package's hoeffd function.

$\endgroup$
6
$\begingroup$

How about this paper:

http://arxiv.org/pdf/0803.4101.pdf

"Measuring and testing dependence by correlation of distances". Székely and Bakirov always have interesting stuff.

There is matlab code for the implementation:

http://www.mathworks.com/matlabcentral/fileexchange/39905-distance-correlation

If you find any other (simple to implement) test for independence let us know.

$\endgroup$
2
  • 2
    $\begingroup$ Welcome to the site, @JLp. We hope to build a permanent repository of high-quality statistical information in the form of questions & answers. As such, one thing we worry about is linkrot. With that in mind, would you mind giving a summary of what is in that paper / how it answers the questions, in case the link were to go dead. It will also help future readers of this thread decide whether they want to invest the time to read the paper. $\endgroup$ Oct 25, 2013 at 15:35
  • $\begingroup$ @gung: this is the same as energy $\endgroup$
    – sds
    Oct 27, 2013 at 17:51
6
$\begingroup$

The link between Distance Covariance and kernel tests (based on the Hilbert-Schmidt independence criterion) is given in the paper:

Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K., Equivalence of distance-based and RKHS-based statistics in hypothesis testing, Annals of Statistics, 41 (5), pp.2263-2702, 2013

It's shown that distance covariance is a special case of the kernel statistic, for a particular family of kernels.

If you're intent on using mutual information, a test based on a binned estimate of the MI is:

Gretton, A. and Gyorfi, L., Consistent Nonparametric Tests of Independence, Journal of Machine Learning Research, 11 , pp.1391--1423, 2010.

If you're interested in getting the best test power, you're better off using the kernel tests, rather than binning and mutual information.

That said, given your variables are univariate, classical nonparametric independence tests like Hoeffding's are probably fine.

$\endgroup$
5
$\begingroup$

Rarely (never?) in statistics can you demonstrate that your sample statistic = a point value. You can test against point values and either exclude them or not exclude them. But the nature of statistics is that it is about examining variable data. Because there is always variance then there will necessarily be no way to know that something is exactly not related, normal, gaussian, etc. You can only know a range of values for it. You could know if a value is excluded from the range of plausible values. For example, it's easy to exclude no relationship and give range of values for how big the relationship is.

Therefore, trying to demonstrate no relationship, essentially the point value of relationship = 0 is not going to meet with success. If you have a range of measures of relationship that are acceptable as approximately 0. Then it would be possible to devise a test.

Assuming that you can accept that limitation it would be helpful to people trying to assist you to provide a scatterplot with a lowess curve. Since you're looking for R solutions try:

scatter.smooth(x, y)

Based on the limited information you've given so far I think a generalized additive model might be the best thing for testing non-independence. If you plot that with CI's around the predicted values you may be able to make statements about a belief of independence. Check out gam in the mgcv package. The help is quite good and there is assistance here regarding the CI.

$\endgroup$
2
$\begingroup$

It may be interesting ...

Garcia, J. E.; Gonzalez-Lopez, V. A. (2014) Independence tests for continuous random variables based on the longest increasing subsequence. Journal of Multivariate Analysis, v. 127 p. 126-146.

http://www.sciencedirect.com/science/article/pii/S0047259X14000335

$\endgroup$
2
1
$\begingroup$

Yet another approach is Constrained Covariance: basically, for a "sufficiently rich" function class $G$, Constrained Covariance of two random variables $X$ and $Y$ is $$ \text{CoCo}(X,Y)=\sup_{g_1,g_2\in G} \text{corr}(g_1(X),g_2(Y)) $$

(possibly related to another answer)

As for the computational aspect, please see Safe, Anytime-Valid Inference (SAVI) and Game-theoretic Statistics (I am omitting the details because they are being published now)

$\endgroup$
2
  • 1
    $\begingroup$ This makes sense and looks promising as an answer, but could you explain how it can be used to assess an independence assumption? $\endgroup$
    – whuber
    Jul 7, 2022 at 15:23
  • $\begingroup$ @whuber: I added a link to SAVI $\endgroup$
    – sds
    Jul 7, 2022 at 18:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.