Return to Answer

I believe it's legitimate to quote the relevant portion of the paragraph in question:

3. The KS test can not be applied in two or more dimensions. Astronomers often have datasets with points distributed in a plane or higher dimensions, rather than along a line. Several papers in the astronomical literature purport to present a two-dimensional KS test, and one is reproduced in the famous volume Numerical Recipes. However, no EDF-based test (this includes KS, AD and related tests) can be applied in two or higher dimensions, because there is no unique way to order the points so that distances between well-defined EDFs can be computed. One can construct a statistic based on some ordering procedure, and then compute the supremum distances between two datasets (or one dataset and a curve). But the critical values of the resulting statistic are not distribution-free.

As stated, this seems too strong.

1. The bivariate distribution function, which is $F(x_1,x_2) = P(X_1\leq x_1,X_2\leq x_2)$ is a map from $\mathbb{R}^2$ to $[0,1]$. That is, the function takes univariate real values between 0 and 1. Those values - being probabilities - are certainly "ordered" already - and this (the value of the function) is the thing we need to make comparisons on for ECDF-based tests. Similarly, the ecdf, $\hat F$ is perfectly well defined in the bivariate case.

I don't think there's necessarily a need to try to turn it into some function of a univariate combined variable as the text suggests. You simply compute $F$ and $\hat F$ at every required combination and compute the difference.

Clearly such a test statistic would not be altered by changes to transformations of the margins, which is to say, if constructed as a test of bivariate independent uniforms, $\mathbf{U}=(U_1,U_2)$, then it works equally well as a test of independent $(X_1,X_2)$ where $U_i=F_i(X_i)$. In that sense, it's distribution-free (we might say 'margin-free').

2. However, there's an underlying point more generally in the broader sense that a naive version of the KS statistic (such as I just described) is not more generally distribution free; we can't simply transform $U$ arbitrarily $X^* = \mathbf{g}(\mathbf{U})$.

In an earlier version of my answer I said:

There's no difficulty, no problem

That's wrong. There are indeed issues if there's a change not just of the margins from bivariate independent uniforms, as just mentioned. However, those difficulties have been considered in several ways in a number of papers that yield bivariate/multivariate versions of Kolmogorov-Smirnov statistics that don't suffer from that problem.

I may come back and add some of those references and some discussion of how they work as soon as time permits.

I believe it's legitimate to quote the relevant portion of the paragraph in question:

3. The KS test can not be applied in two or more dimensions. Astronomers often have datasets with points distributed in a plane or higher dimensions, rather than along a line. Several papers in the astronomical literature purport to present a two-dimensional KS test, and one is reproduced in the famous volume Numerical Recipes. However, no EDF-based test (this includes KS, AD and related tests) can be applied in two or higher dimensions, because there is no unique way to order the points so that distances between well-defined EDFs can be computed. One can construct a statistic based on some ordering procedure, and then compute the supremum distances between two datasets (or one dataset and a curve). But the critical values of the resulting statistic are not distribution-free.

As stated, this seems too strong.

1. The bivariate distribution function, which is $F(x_1,x_2) = P(X_1\leq x_1,X_2\leq x_2)$ is a map from $\mathbb{R}^2$ to $[0,1]$. That is, the function takes univariate real values between 0 and 1. Those values - being probabilities - are certainly "ordered" already - and this (the value of the function) is the thing we need to make comparisons on for ECDF-based tests. Similarly, the ecdf, $\hat F$ is perfectly well defined in the bivariate case.

I don't think there's necessarily a need to try to turn it into some function of a univariate combined variable as the text suggests. You simply compute $F$ and $\hat F$ at every required combination and compute the difference.

2. However, on the question of whether it's distribution-free, they have a point:

a) clearly such a test statistic would not be altered by changes to transformations of the margins, which is to say, if constructed as a test of bivariate independent uniforms, $\mathbf{U}=(U_1,U_2)$, then it works equally well as a test of independent $(X_1,X_2)$ where $U_i=F_i(X_i)$. In that sense, it's distribution-free (we might say 'margin-free').

b) however, there's an underlying point more generally in the broader sense that a naive version of the KS statistic (such as I just described) is not more generally distribution free; we can't simply transform $U$ arbitrarily $X^* = \mathbf{g}(\mathbf{U})$.

In an earlier version of my answer I said:

There's no difficulty, no problem

That's wrong. There are indeed issues if there's a change not just of the margins from bivariate independent uniforms, as just mentioned. However, those difficulties have been considered in several ways in a number of papers that yield bivariate/multivariate versions of Kolmogorov-Smirnov statistics that don't suffer from that problem.

I may come back and add some of those references and some discussion of how they work as soon as time permits.

I believe it's legitimate to quote the relevant portion of the paragraph in question:

3. The KS test can not be applied in two or more dimensions. Astronomers often have datasets with points distributed in a plane or higher dimensions, rather than along a line. Several papers in the astronomical literature purport to present a two-dimensional KS test, and one is reproduced in the famous volume Numerical Recipes. However, no EDF-based test (this includes KS, AD and related tests) can be applied in two or higher dimensions, because there is no unique way to order the points so that distances between well-defined EDFs can be computed. One can construct a statistic based on some ordering procedure, and then compute the supremum distances between two datasets (or one dataset and a curve). But the critical values of the resulting statistic are not distribution-free.

As stated, this seems too strong.

1. The bivariate distribution function, which is $F(x_1,x_2) = P(X_1\leq x_1,X_2\leq x_2)$ is a map from $\mathbb{R}^2$ to $[0,1]$. That is, the function takes univariate real values between 0 and 1. Those values - being probabilities - are certainly "ordered" already - and this (the value of the function) is the thing we need to make comparisons on for ECDF-based tests. Similarly, the ecdf, $\hat F$ is perfectly well defined in the bivariate case.

I don't think there's necessarily a need to try to turn it into some function of a univariate combined variable as the text suggests. You simply compute $F$ and $\hat F$ at every required combination and compute the difference.

Clearly such a test statistic would not be altered by changes to transformations of the margins, which is to say, if constructed as a test of bivariate independent uniforms, $\mathbf{U}=(U_1,U_2)$, then it works equally well as a test of independent $(X_1,X_2)$ where $U_i=F_i(X_i)$. In that sense, it's distribution-free (we might say 'margin-free').

However, there's an underlying point right more generally in the broader sense that a naive version of the KS statistics is not more generally distribution free; we can't simply transform $U$ arbitrarily $X^* = \mathbf{g}(\mathbf{U})$.

In an earlier version of my answer I said:

There's no difficulty, no problem

That's wrong. There are indeed issues if there's a change not just of the margins from bivcariate independent uniforms, as just mentioned. However, those difficulties have been considered in several ways in a number of papers that yield bivariate/multivariate versions of Kolmogorov-Smirnov statistics that don't suffer from that problem.

I may come back and add some of those references and discussion as soon as time permits.

I believe it's legitimate to quote the relevant portion of the paragraph in question:

3. The KS test can not be applied in two or more dimensions. Astronomers often have datasets with points distributed in a plane or higher dimensions, rather than along a line. Several papers in the astronomical literature purport to present a two-dimensional KS test, and one is reproduced in the famous volume Numerical Recipes. However, no EDF-based test (this includes KS, AD and related tests) can be applied in two or higher dimensions, because there is no unique way to order the points so that distances between well-defined EDFs can be computed. One can construct a statistic based on some ordering procedure, and then compute the supremum distances between two datasets (or one dataset and a curve). But the critical values of the resulting statistic are not distribution-free.

As stated, this seems too strong.

1. The bivariate distribution function, which is $F(x_1,x_2) = P(X_1\leq x_1,X_2\leq x_2)$ is a map from $\mathbb{R}^2$ to $[0,1]$. That is, the function takes univariate real values between 0 and 1. Those values - being probabilities - are certainly "ordered" already - and this (the value of the function) is the thing we need to make comparisons on for ECDF-based tests. Similarly, the ecdf, $\hat F$ is perfectly well defined in the bivariate case.

I don't think there's necessarily a need to try to turn it into some function of a univariate combined variable as the text suggests. You simply compute $F$ and $\hat F$ at every required combination and compute the difference.

Clearly such a test statistic would not be altered by changes to transformations of the margins, which is to say, if constructed as a test of bivariate independent uniforms, $\mathbf{U}=(U_1,U_2)$, then it works equally well as a test of independent $(X_1,X_2)$ where $U_i=F_i(X_i)$. In that sense, it's distribution-free (we might say 'margin-free').

2. However, there's an underlying point more generally in the broader sense that a naive version of the KS statistic (such as I just described) is not more generally distribution free; we can't simply transform $U$ arbitrarily $X^* = \mathbf{g}(\mathbf{U})$.

In an earlier version of my answer I said:

There's no difficulty, no problem

That's wrong. There are indeed issues if there's a change not just of the margins from bivariate independent uniforms, as just mentioned. However, those difficulties have been considered in several ways in a number of papers that yield bivariate/multivariate versions of Kolmogorov-Smirnov statistics that don't suffer from that problem.

I may come back and add some of those references and some discussion of how they work as soon as time permits.

This is simply wrong. The bivariate distribution function, which is $F(x_1,x_2) = P(X_1\leq x_1,X_2\leq x_2)$ is a map from $\mathbb{R}^2$ to $[0,1]$. That is, the function takes univariate real values between 0 and 1. Those values are certainly "ordered" already - and this (the value of the function) is the thing we need to make comparisons on for ECDF-based tests.

Similarly, the ecdf, $\hat F$ is perfectly well defined in the bivariate case.

There's no need to "order" the bivariate $X$ values*. You simply compute $F$ and $\hat F$ at every required combination and compute the difference. There's no need to try to turn it into some function of a univariate combined variable. Without that unnecessary step, the test remains (demonstrably) distribution-free.

* at least not beyond the sense of the partial order on subsets of the X-space already used by the definition of cumulative distribution functions.

(My bivariate comments carry over to higher-dimension X-vectors as well, but let's stick with the bivariate case for the moment.)

There's no difficulty, no problem, for all their assertions to the contrary.

I'd be surprised if there's not (eventually) a retraction of the paper from the site.

That they have put up a paper with an error in it is not a particular problem, necessarily. People make mistakes, and other people pick them up. This would not survive competent peer review -- indeed that's the point of peer review. Not everything that gets put up on a university web site has been through that process, and errors are to be expected (and sometimes occur even with that process). I know I've had working papers that had gaps or small error; other people read them and ask questions, and they get sorted out.

In this case, that misplaced criticism is kind of the point of the paper, though, so it will probably simply be dropped as there's not much left over to rescue. That happens too.

[We should also note that this doesn't appear to be even a working paper on the way to a formal submission to a journal, but simply something they put up, perhaps as a warning to their colleagues; it may never be subject to any formal review process. I note that the forum they suggest for discussion of the paper only has a couple of posts for this whole year. Something that inactive isn't going to be a source of many critical eyes, and quite possibly has few people with much formal background in statistics, so the necessary nudge they'll need to reconsider their position may not some as quickly as one might hope. I'd happily take it up with them on our chat forum, were they inclined to do so.]

As stated, this seems too strong.

1. The bivariate distribution function, which is $F(x_1,x_2) = P(X_1\leq x_1,X_2\leq x_2)$ is a map from $\mathbb{R}^2$ to $[0,1]$. That is, the function takes univariate real values between 0 and 1. Those values - being probabilities - are certainly "ordered" already - and this (the value of the function) is the thing we need to make comparisons on for ECDF-based tests. Similarly, the ecdf, $\hat F$ is perfectly well defined in the bivariate case.

I don't think there's necessarily a need to try to turn it into some function of a univariate combined variable as the text suggests. You simply compute $F$ and $\hat F$ at every required combination and compute the difference.

Clearly such a test statistic would not be altered by changes to transformations of the margins, which is to say, if constructed as a test of bivariate independent uniforms, $\mathbf{U}=(U_1,U_2)$, then it works equally well as a test of independent $(X_1,X_2)$ where $U_i=F_i(X_i)$. In that sense, it's distribution-free (we might say 'margin-free').

However, there's an underlying point right more generally in the broader sense that a naive version of the KS statistics is not more generally distribution free; we can't simply transform $U$ arbitrarily $X^* = \mathbf{g}(\mathbf{U})$.

In an earlier version of my answer I said:

There's no difficulty, no problem

That's wrong. There are indeed issues if there's a change not just of the margins from bivcariate independent uniforms, as just mentioned. However, those difficulties have been considered in several ways in a number of papers that yield bivariate/multivariate versions of Kolmogorov-Smirnov statistics that don't suffer from that problem.

I may come back and add some of those references and discussion as soon as time permits.