17
$\begingroup$

A well-known rule of thumb is that for high dimensions $d$, the Gaussian distribution $N(0,I_d)$ is approximated by the uniform distribution on a sphere $U_{\sqrt{d}S^{d-1}}$. This has been mentioned on previous cross validated posts, e.g. Why is Gaussian distribution on high dimensional space like a soap bubble. This can be made precise using techniques from large deviations theory (e.g. theorem 2.9 in https://www.cs.cornell.edu/jeh/book.pdf).

Is there an analogous picture for a high-dimensional Cauchy distribution? By a high-dimensional Cauchy, I mean a random vector in which each component is independently distributed according to a 1D unit Cauchy.

Do the samples accumulate near some manifold, and can we bound the probability of deviation from the manifold? Any references are appreciated .

$\endgroup$
15
  • 2
    $\begingroup$ Which high-dimensional Cauchy distribution are you referring to? $\endgroup$
    – whuber
    Commented Nov 27, 2021 at 18:04
  • $\begingroup$ Do you mean $d$ independent $t_1$ distributions? $\endgroup$
    – Dave
    Commented Nov 27, 2021 at 18:05
  • 1
    $\begingroup$ @markowitz Would $iid$ Cauchy variables fit that criterion of "every linear combination of its components" being Cauchy? The sum of two $iid$ Cauchy variables is Cauchy. $\endgroup$
    – Dave
    Commented Nov 29, 2021 at 17:26
  • 1
    $\begingroup$ Why would that not be one particular multivariate Cauchy the way that independent Gaussian variables is one particular multivariate Gaussian? $\endgroup$
    – Dave
    Commented Nov 29, 2021 at 17:37
  • 1
    $\begingroup$ I wanted only underscore that the question can be missinterpreted because multivariate Cauchy (at least in standard definition) and multivariate distribution with independent Cauchy marginals are not equivalent. $\endgroup$
    – markowitz
    Commented Nov 30, 2021 at 7:59

1 Answer 1

12
+50
$\begingroup$

The main challenge in this question lies in interpreting the sense of "accumulate around some manifold." The difficulty is that no such thing can happen, because as the vector length $d$ grows, the vector does not not even stay in the same space!

Interpreting the question

We need therefore to formulate a concept of a "sequence of manifolds" in the nested vector spaces $\mathbb{R}^0 \subset \mathbb{R}^1 \subset \mathbb{R}^2 \subset \cdots \subset \mathbb{R}^d \subset \cdots.$ Looking to the Normal distribution for guidance, that sequence would be something like the sequence of spheres $S^{d-1}(\sqrt d)$ defined by

$$\mathbb{R}^d \supset S^{d-1}(\sqrt{d}) = \{(x_1,x_2,\ldots, x_d)\mid x_1^2+x_2^2+\cdots + x_d^2=d\}$$

for $d=0,1,2,\ldots.$

Because accumulation of points near spheres is largely a geometric property of Euclidean distances in high-dimensional spaces, we ought first to wonder whether spheres will continue to work for Cauchy variables. Imagine a sequence of iid such variables $\mathbf X = X_1, X_2, \ldots, X_d, \ldots.$ In the standard Normal case, the distribution of

$$R_d^2(\mathbf X) = X_1^2 + X_2^2 + \cdots + X_d^2$$

is easy to study because it has a chi-squared distribution of $d$ degrees of freedom and elementary analysis shows this distribution has an expectation of $d$ and a variance of $2d.$ Thus $R_d(\mathbf X)$ has a Chi distribution which, for large $d,$ has a mean close to $\sqrt{d-1/2}$ and a standard deviation close to $\sqrt{1/2}.$ If we take $S^{d-1}(\sqrt{d-1/2})$ to be the approximating sphere, this standard deviation grows small relative to the sphere's radius. It is in this sense we can say that the sequence of vectors $(X_1,X_2,\ldots, X_d)$ approximates the sequence of manifolds $S^{d-1}(\sqrt{d-1/2}).$ Finally, since asymptotically these manifolds are arbitrarily close to $S^{d-1}(\sqrt{d}),$ we obtain the assertion in the question.

One important conclusion is that the sequence of approximating manifolds will not be unique.

First attempt at a solution

Emulating this approach, we would like to learn about the distribution of $R^2_d(\mathbf X)$ when the $X_i$ are iid with common Cauchy distributions. The Cauchy distribution has a positive density $$f(x)=\frac{1}{\pi(1+x^2)}$$ at any real number $x.$ Unfortunately that immediately implies that no power of $|X_i|$ of $1$ or greater has a finite expectation. In particular, $E[R^2_d(\mathbf X)]=\infty$ and $\operatorname{Var}(R^2_d(\mathbf X))=\infty.$

It nevertheless is possible for the samples to accumulate along a sequence of manifolds, provided a decreasing proportion of the components grow very large (to make the moments infinite). Rather than wading through difficult integral estimates, let's just take a quick look at some quantiles of these distributions. To make them all comparable across a large range of $d,$ I have examined $R_d(\mathbf X)/\sqrt d$ (which in the Normal case converges to unity).

The figure plots selected quantiles of these distributions for an interesting collection of distributions. The Cauchy is at the left, some other (suitably scaled) Student t distributions are next, then followed by the standard Normal and the Uniform$(-2,2)$ distribution. I have scaled the $t(4),$ Normal, and Uniform distributions to yield asymptotic values of $R^d$ approaching $1.$

Figure 1

(Notice the log-log scales.)

These distributions fall into two groups. The two at the left have infinite variances. The three at the right have finite variances, which means strong laws of averages (as well as the Central Limit Theorem) apply. The CLT informs us (1) the expectation of $R^d$ will converge and (2) its variance will shrink asymptotically to zero (at a rate proportional to $1/d$). You can see both these things occurring in the plot: from left to right in the panels, the quantiles converge to a common level value.

This convergence does not happen in the left two panels. Because the Student t(2) distribution has a finite expectation, the quantiles level off--but it looks like they might not all converge to the same value. At the left, because the Cauchy distribution has infinite expectation and infinite variance, the quantiles grow ever larger (according to some power law, it appears) and they grow exponentially further apart.

There is no collection of approximating spheres for the Cauchy distribution.

Second attempt at a solution

How, then, should be proceed? A natural way to interpret the question is to re-ask it a little more broadly:

Where do most of the data tend to be in large datasets?

The answer, of course, is they will be located where the joint distribution has the greatest probability density. Equivalently, the log density will be greatest. At the point $\mathbf x = (x_1,x_2,\ldots,x_d)\in\mathbb{R}^d,$ the independence of the $X_i$ implies the log density is the sum of logs of the individual (marginal) densities,

$$\log F_{{\mathbf X}_d}(\mathbf x) = -d\log(\pi) - \sum_{i=1}^d \log(1 + x_i^2).$$

Now, the previous simulations suggest that even very low percentiles of these samples grow arbitrarily large as $d$ increases. That is, for sufficiently large $d,$ almost all the components of $(X_1,X_2,\ldots, X_d)$ get pretty large. By that I mean in estimating $\log(1+x_i^2),$ we will do fine by ignoring the $1+$ term, because it's usually much smaller than $x_i^2$ for large $d.$ Thus, for huge $d,$

$$\log F_{{\mathbf X}_d}(\mathbf x) \approx -d\log(\pi) -2\sum_{i=1}^d 2\log(|x_i|).$$

This tells us the level sets of $F_{{\mathbf X}_d}$ will be those where the sum of logarithms of the $x_i$ is a constant. Equivalently,

For large $d,$ the level sets ("contours") of the joint distribution of $(X_1,X_2,\ldots, X_d)$ are well approximated by the manifolds defined by $x_1x_2\cdots x_d = \text{Constant}.$

How are these contour levels distributed? Let's examine another simulation. As before, to make results for widely differing values of $d$ comparable, I have tracked $\log F_{{\mathbf X}_d}(\mathbf x)/d,$ which is the average of the densities of the components. Recall that such an average will asymptotically approximate the expectation: that is,

$$\log F_{{\mathbf X}_d}(\mathbf x)/d \approx 2\int_0^\infty \log(f(x)) f(x)\,\mathrm{d}x = \int_{-\infty}^\infty \log(f(x)) f(x)\,\mathrm{d}x$$

(because $f$ is symmetric about $0$). This is, of course, the negative of the Entropy of the Cauchy distribution. It equals $\log(4\pi) \approx 2.53.$

The figure shows histograms of $\log F_{{\mathbf X}_d}(\mathbf x)/d$ for 10,000 independent samples of each indicated sample size from $d=1$ to $d=2500.$

Figure 2

Not only do the histograms approach a Normal distribution shape, they also shrink around their common expectation of $\log(4\pi)$ (shown as vertical blue bars). Look at the scales: for $d=1$ the range of values runs from almost $0$ to about $20.$ For $d=2500$ the range has shrunk steadily to the interval $[2.40, 2.70].$

This behavior looks like the Central Limit Theorem in action--and it is. The sequence of random variables $\log f(X_1), \log f(X_2), \ldots, \log f(X_d), \ldots$ is iid and each of these variables has finite expectation and finite variance. (That is a matter of computing two integrals for the first and second moments.)

Let's connect this back to the Normal case, where $\log f (x) = -x^2/2 - \log(2\pi)/2.$ The corresponding level curves are of the form $x_1^2 + \cdots + x_d^2 = \text{Constant},$ implicating the spheres (of suitably varying radius) as the sequence of approximating manifolds.

Because these results are all intended to be asymptotic, we could simplify matters a bit and propose that

the sequence of approximating manifolds can be chosen as the quasi-hyperboloids of the form $x_1x_2\cdots x_d = C(d)$ where $C(d)$ depends (in a readily computable way) on the entropy and $d.$

Here's a picture for $d=2.$

Figure 3

The colors (and labels) denote values of $\log F_{\mathbf X_2}.$ The gray curves are various level sets. The arrows point "downhill" towards smaller values of the joint density: they help show the shapes of these manifolds better.

You can already see how many of the level sets are "trying" to approximate hyperboloids of the form $x_1x_2=\text{Constant}.$ They don't do this too well near the center, where most of the density is. That doesn't matter: $d=2$ just isn't quite large enough to characterize asymptotic behavior! Nevertheless, this plot is very suggestive and sketches how things must look (qualitatively) in much higher dimensions.

In particular, notice how these level curves "spike outwards" along the coordinate planes (axes in 2D). Herein lies the resolution of an apparent paradox: if Cauchy samples must diverge rapidly from any sequence of spheres (as shown above), how can they possibly accumulate around any sequence of manifolds whatsoever? The answer is that those manifolds must diverge. Indeed, in 2D the approximating manifolds I have proposed, $x_1 x_2=\text{Constant},$ are all unbounded hyperbolae. The same holds in higher dimensions.


All these results (and much more) are routinely stated and demonstrated in a different form in courses on asymptotics and Maximum Likelihood. The present setting differs only in the focus on the question of what large iid samples "look like" when considered geometrically as vectors.

$\endgroup$
4
  • $\begingroup$ I have trouble with this explanation. Wouldn't the same argument suggest that (zero-mean, independent) Gaussians concentrate around zero instead of a shell with radius $\sqrt{d}$? Isn't it also necessary to account for the fact that the volume of spheres of constant radius goes to zero for increasing dimension? I believe my trouble stems from the statement "they will be located where the joint distribution has the greatest probability density" instead of something along the lines of "Where the product of density and volume is greatest". $\endgroup$
    – g g
    Commented Dec 4, 2021 at 10:34
  • 1
    $\begingroup$ @gg That is exactly what I was wrestling with. All I can recommend is follow the mathematics. The simulations will help with the intuition. Concerning your first remark: I cannot see how any part of this answer would suggest Gaussians concentrate at $0.$ It might help to study plots of the densities of the chi-square distributions describing the squared distances of the Gaussian samples (from the origin): for $d\gt 2,$ they ascribe zero density to the origin and a peak density at a squared distance of $d-2.$ $\endgroup$
    – whuber
    Commented Dec 4, 2021 at 14:29
  • $\begingroup$ @whuber Great answer, thanks! A few questions: (1) should the hyperboloid be defined using an absolute value instead $|x_1\dots x_d|=C(d)$? (2) The argument seems to rest upon applying the CLT to the sequence of transformed variables $g(X_i)$. But supposing I considered a different transformation,e.g. $g(x)=|x|^.5$? (Consider the Gaussian case for simplicity). Applying the CLT, we see that $d^{-1}\sum_{i=1}^d |X_i|^.5$ has mean $O(1)$ and standard deviation $O(d^{-.5})$, (1/2) $\endgroup$ Commented Dec 5, 2021 at 14:48
  • $\begingroup$ therefore samples from the Gaussian concentrate near the ''manifold" $\{\sum_i |x_i|^{.5}=C(d)\}$, which looks quite different from the sphere. So why does the argument work for $g(x)=x^2$ but not $|x|^.5$? Presumably it has to do with the fact that in the former case, the equiprobability regions of $\sum_i g(X_i)$ coincide with the level sets of the density function of $(X_1, \dots, X_d)$, but I don't quite see how or whether this fact is used. (2/2) $\endgroup$ Commented Dec 5, 2021 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.