19
$\begingroup$

In this famous post "Gaussian Distributions are Soap Bubbles" it is claimed that the distribution of the points looks like a soap bubble (where it is less dense in the center and more dense at the edge) instead of a bold of mold where it is more dense in the center. I would expect that it is more dense in the center like it is in two or three dimensional.
From the post, I could not figure out why this is the case. It uses three figures which I could not figure out what they are telling.
enter image description here Can someone clarify this figures and the main question why it is looking like a soap bubble in high dimensions?

$\endgroup$
6
  • 3
    $\begingroup$ The mode is always at the center (irrespective the dimension), but in high dimensions it is very little probability close to the mode, the mass of probability is further out. I will try to add a more detailed answer! $\endgroup$ Jul 27, 2019 at 10:42
  • 2
    $\begingroup$ There is no edge to a Gaussian distribution... I personally think that this is a terrible analogy, not only for that reason. $\endgroup$
    – jbowman
    Jul 27, 2019 at 14:54
  • 3
    $\begingroup$ This is not special to Gaussian distributions: it applies to every distribution of finite variance. The squared radius is the sum of iid variables and the Central Limit Theorem takes care of the rest. What is special about Gaussian distributions is that the soap is distributed uniformly in the angular directions. $\endgroup$
    – whuber
    Jul 27, 2019 at 18:21
  • 1
    $\begingroup$ This is just the curse of dimensionality, isn't it? $\endgroup$
    – Akababa
    Jul 27, 2019 at 21:12
  • 1
    $\begingroup$ @whuber "The squared radius is the sum of iid variables...." If I recall correctly, both spherical symmetry of the pdf and independence of the random variables occurs only in the Gaussian case? If so, the random variables being squared are identically distributed, but not independent (what I call NIBNID in my answer) in all cases except when the random variables are Gaussian. $\endgroup$ Jul 27, 2019 at 22:23

5 Answers 5

19
$\begingroup$

I can't answer about what the OP's famous post claims, but let us consider the simpler case of uniform distributions on the unit disc: $(X,Y)$ is uniformly distributed on the unit disc (that is, $f(X,Y)(x,y) = \frac 1\pi$ for $x^2+y^2 < 1$. What is the probability that $(X,Y)$ is closer to the unit circle, that is, closer to the boundary of the unit disc than it is to the origin (center of the circle)? Well, only those points that lie inside the circle of radius $\frac 12$ are at distance $< \frac 12$ from the origin, and so all points outside this smaller circle are at distance $> \frac 12$ from the origin. It is an easy computation to arrive at $$P\left(\frac 12 < \sqrt{X^2+Y^2} < 1\right) = 1- P\left(0\leq \sqrt{X^2+Y^2} < \frac 12\right) = 1 - \frac 1\pi \cdot \pi\left(\frac 12\right)^2 = \frac 34.$$ A similar calculation for a uniform distribution on the interior of a unit sphere in 3 dimensions (the pdf has value $\frac{3}{4\pi}$ on the interior) gives \begin{align} P\left(\frac 12 < \sqrt{X^2+Y^2+Z^2} < 1\right) &= 1- P\left(0\leq \sqrt{X^2+Y^2+Z^2} < \frac 12\right)\\ &= 1 - \frac{3}{4\pi} \cdot \frac{4\pi}{3}\left(\frac 12\right)^3\\ &= \frac 78. \end{align} Generalizing to $n > 3$ dimensions and remembering that the volume of an $n$-dimeensional hypersphere or radius $r$ is proportional to $r^n$, we get by very similar calculations that $$P\left(\frac 12 < \sqrt{\sum_{i=1}^n X_i^2} < 1\right) = \frac{2^n-1}{2^n},$$ that is, most of the probability mass_ lies closer to the surface of the sphere than to the origin. As a final comment, note that the $X_i$ are NIBNID random variables which acronym stands for Not Independent But Nonetheless Identically Distributed.

Turning to IID standard Gaussian random variables, the joint density is not uniformly distributed but has a very pronounced peak at the origin. But, there is so little volume near the center of a hypersphere as compared to closer to the surface that when we integrate the density over the volume of a hypersphere of small radius $r$ to find $P\left(\sqrt{\sum_{i=1}^n X_i^2} < r\right)$, most of this probability mass is obtained from the small contributions from the periphery (there are so many of them) and very little from the few but larger contributions from the core; that is, most of the probability mass lies closer to the skin of the orange than to the center. But things change as $r$ increases. Since $\sum_{i=1}^n X_i^2$ is a $\chi^2$ random variable with $n$ degrees of freedom (with mean $n$ and variance $2n$), which for large $n$ can be approximated as a Gaussian random variable with the same mean and variance) most of its probability was lies in the range $\left[n-\sqrt{18n},n-\sqrt{18n}\right] = [\mu-3\sigma,\mu+3\sigma]$. Put another way, the quantity $P\left({\sum_{i=1}^n X_i^2} < r^2\right)$ is close to $0$ for small $r$ (the nearly empty space inside the soap bubble), and then (regarded as a function of $r$) increases very rapidly with $r$ in the close vicinity of $r=\sqrt n$ (this is the thin skin of the bubble where most of the mass is) to almost $1$, and then very slowly to its asymptotic value of $1$ (the nearly empty space outside the bubble). In short, the soap bubble analogy is very apt for Gaussian distributions; almost all the probability mass of the joint pdf of $n$ standard Gaussian random variables does indeed lie in a very thin shell of radius $\approx \sqrt n$ and there is very little probability mass that is not in the shell -- both the interior and the exterior of the shell is mostly empty as is the case with soap bubbles.

$\endgroup$
1
  • 3
    $\begingroup$ Very sweet answer (+1)! By the way, the volume of an n-dimensional sphere, of radius r, is $$ \pi ^{n/2}r^n / \Gamma [(n/2)+1] $$. $\endgroup$
    – Ed V
    Jul 27, 2019 at 16:46
7
$\begingroup$

The post you link to concerns the use of the normal distribution in high-dimensional problems. So, suppose you are working in a space $\mathbb{R}^m$ where the dimension $m$ is large. Let $\boldsymbol{I}$ be the $m$-dimensional identity matrix and consider a normal random vector:

$$\mathbf{X} \equiv (X_1,...,X_m) \sim \text{N}(\mathbf{0}, \sigma^2 \boldsymbol{I}).$$

A well-known property of this distribution is that a centered and normed normal random vector is uniformly distributed on the unit sphere. That is, if we let $\mathcal{S}_r^m \equiv \{ \mathbf{x} \in \mathbb{R}^m | \sum x_i^2 = r^2 \}$ denote the $m$-dimensional sphere with radius $r$, then we have:

$$\frac{\mathbf{X}}{||\mathbf{X}||} \sim \text{U}(\mathcal{S}_1^m).$$

It is also well-known that the distribution of the scaled-norm of the random vector is:

$$\frac{||\mathbf{X}||}{\sigma \sqrt{m}} \sim \frac{\chi_m}{\sqrt{m}}.$$

Taking $m \rightarrow \infty$, the right-hand-side convergences in probability to one. Thus, for large $m$ we have:

$$\mathbf{X} \overset{\text{Approx}}{\sim} \text{U}(\mathcal{S}_{\sigma \sqrt{m}}^m)$$

This shows that when $m$ becomes large, the points from this normal random vector are approximately distributed on the surface of a unit sphere with radius $\sigma \sqrt{m}$. This is what the linked post is referring to when it notes that "...in high dimensions, Gaussian distributions are practically indistinguishable from uniform distributions on the unit sphere".

$\endgroup$
0
$\begingroup$

I really think that the vision of an empty bubble is misleading.
(tl-dr: Instead of an empty bubble I think is better to say that it resembles to a star with n-vertices where $n\rightarrow\infty$, or some kind of a non empty fractal structure with the length of its border going to $\infty$). But still more dense in the center.

The higher the dimension is, there are more points close to the border of a n-sphere than to its center (easy to see moving from 2D to 3D), but for a multivariate normal distribution: $$P\left( \left\{ \text{n-sphere center in} \left(0,...,0\right) \text{ and } r=\delta \right\}\right ) > P\left( \left\{ \text{n-sphere center in} \left(\delta,...,0\right) \text{ and } r=\delta \right\}\right )$$ Always, for any dimension, for any $\delta$, the probability is higher the closer to the center (for similar areas of course). So: is still more dense in the center, but is has less center (center is never 0 because remember $dimension \rightarrow \infty$ and not $dimension=\infty$).

Yes... if you pick a point at random is more likely to be in the border than in the center, but not because the probability in the center is "empty as a bubble", but just becasue there are more points close to the border.
This already happens in 2D: in a 2D-Sphere of radius=1 "there are" $\pi/4$ points at distance ½ of the center, while there are $3/4*\pi$ points at distance less than ½ of the border. And actually with a bivariate normal distribution the probability of picking a point next to the border is higher than picking one in the center (see code below).
Another way to visualize it is comparing a 2D sphere vrs a 2D star, as the star has “More border” the probability of picking a point next the border in a star is higher, but the start is not “empty”.

Instead of an empty bubble It will be better to say: it resembles to a star with n-vertices where $n\rightarrow\infty$, or some kind of a non empty fractal structure with the length of its border going to $\infty$. But still more dense in the center.

PD: Using r

library(shotGroups)
 
#using 2d multivariate normal distribution

#probability of choosing a point inside the cercle with radius delta=0.5 
inner_cercle_prob <- pmvnEll(r=0.5, sigma=diag(2), mu=c(0,0), e=diag(2), x0=c(0,0))

#probabiliyt of choosing a point inside the cercle of radius=1
full_cercle_prob <- pmvnEll(r=1, sigma=diag(2), mu=c(0,0), e=diag(2), x0=c(0,0))
  
#probability of choosin a point inside the cercle but closer to the border
corona_prob=full_cercle_prob-inner_cercle_prob
 
#probability of choosin a point outside the cercle 
outside_cercle_prob=1-full_cercle_prob
 
outside_cercle_prob
[1] 0.6065307
corona_prob
[1] 0.2759662
inner_cercle_prob
[1] 0.1175031

#but for cercles with same radius, the one closer to the center as higher prob.
pmvnEll(r=0.5, sigma=diag(2), mu=c(0,0), e=diag(2), x0=c(0,0))
[1] 0.1175031
pmvnEll(r=0.5, sigma=diag(2), mu=c(0,0), e=diag(2), x0=c(0.5,0))
[1] 0.1044914
$\endgroup$
1
  • 1
    $\begingroup$ These descriptions of a "star" are clearly incorrect, because multivariate Normals are elliptical distributions. In particular, the standard multivariate Normal distribution is perfectly spherically symmetric: no direction is distinguished. That's the very opposite of a star. $\endgroup$
    – whuber
    Aug 19, 2021 at 16:03
0
$\begingroup$

This is an old post with some great responses but I'd like to give a different perspective.

Assume we take a sample $x$ from $\mathcal{N}(\vec0, \mathcal{I})$ in $D$ dimensions. If the high-dimensional Gaussian is hollow, then that would mean at least one coordinate of our sample $x$ deviates from the mean. By the CDF of the normal distribution, the chance of $x$'s first coordinate being within one standard deviation is about $~68.2\%$. Now what is the chance that both the first and second coordinates are within one standard deviation? They are independent, so it's $0.682^2$. By extension, the probability that a sample in $D$-dimensional space is within one standard deviation along every axis is $0.68^D$. Naturally, this goes to 0 very quickly.

$\endgroup$
2
  • 1
    $\begingroup$ This is an interesting perspective, but it seems to carry a basic flaw. You can do the same calculation for any set of intervals, rather than the 1 sd interval around 0. If you make these intervals all of length 2 sds, the chance that the sample will lie within their Cartesian product can never exceed $(0.682)^D.$ In this sense you seem to have shown that the density is greatest at $0:$ namely, $0=(0,0,\ldots,0)$ remains the mode of the distribution. Indeed, this is obvious when you inspect any formula for the density. Thus, your remarks only deepen the apparent paradox. $\endgroup$
    – whuber
    Oct 6, 2021 at 13:04
  • $\begingroup$ Hmm, I think I am either missing your point or I disagree. I am saying that each coordinate has a chance of being either within or outside one standard deviation (1 std. dev. is arbitrary of course). Thus, we end up with a binomial distribution of sorts. The odds that all of your coordinates are within one standard deviation is '$D$ choose 0', where $D$ is the dimensionality of the space. So we are most likely to see points at distance $\frac{D}{2} || \sigma ||$, or whatever the most likely element is under the binomial distribution. $\endgroup$
    – ness_boy
    Oct 6, 2021 at 20:03
-1
$\begingroup$

I don't think it is true that "a Gaussian distribution in higher dimensions looks like a soap bubble". But first let's see why, in accordance with some of the very detailed responses above, one might be led to think so.

In Cartesian coordinates in $D$ dimensions, after standardization, the probability density looks like $$ p(\vec{X}) ~d\vec{X} \sim V^{-D/2} \exp(-\frac{||\vec{X}||}{2V}) ~d\vec{X}$$ where $V$ is the 1D variance of each variable.

We can rewrite this in spherical coordinates, and use spherical symmetry to integrate over the D-1 dimensional spheres. I'll leave out the factor corresponding to the volume of the D-1 sphere (not the ball which is the "interior" of the sphere). The radial distribution is $$ p(r)dr \sim V^{-\frac{D}2} r^{D-1} \exp(-\frac{r^2}{2V})~dr = V^{-\frac{D}2} r^{D-2} \exp(-\frac{r^2}{2V}) ~ \frac12 dr^2, ~r > 0 $$ Introducing $z = \frac{r^2}2$, the distribution is: $$p(z)dz \sim V^{-\frac{D}2} z^{\frac{D}2-1} \exp(-\frac{z}{V})~dz$$ which is just a Gamma Distribution (in $z$). Now you can look up https://en.wikipedia.org/wiki/Gamma_distribution or calculate the mean radius (modulo some irrelevant factors and $\pm 1$): $$<r> \sim D \sqrt{V}$$ and the variance of the radius $<(r-<r>)^2>$: $$var(r) \sim DV$$ which means the standard deviation of $r$ $$SD(r) \sim\sqrt{DV}$$. The relative SD: $$ \frac{SD(r)}{<r>} \sim \frac1{\sqrt{D}}$$ which, as the number of dimensions $D$ increases, tends to 0.

So we think that looks like a bubble. But, the question is, as a distribution, does the radial distribution $\rightarrow \delta(r-r_0)$ in the limit $D\rightarrow\infty$?

Now let's go back to that D-dimensional distribution in Cartesian coordinates. It is peaked at the origin and falls off as your distance from the origin increases. It looks nothing like a bubble. If you had a mass density distributed like this you would not encounter any bubble that you have to pierce nor any "thickening" at $r_0$, in fact the density would continue to increase as you travelled towards the center. It is only when we integrate over the shells of fixed radius and collapse them to a single radial point do we get the relative standard deviation tending to 0 with increased dimensionality.

So, no, it is not true that "a Gaussian distribution in higher dimensions looks like a soap bubble".

$\endgroup$
10
  • $\begingroup$ Re " let's go back to that D-dimensional distribution in Cartesian coordinates:" exactly what are you referring to? If you mean the distribution with density $p,$ then you are flatly contradicting the result of your preceding (correct) analysis. $\endgroup$
    – whuber
    May 23, 2020 at 15:06
  • 1
    $\begingroup$ Well, if we take it very literally, sure, it doesn't look much like a soap bubble, as the density is actually quite low, but the surface area of the shell is enormous. But the reason for the analogy is pretty clear: a projection down to 3D where shells have the same density by radius produces the soap bubble. Ultimately it's only "wrong" because there is simply no visualizable analogy for extremely high dimensional space. $\endgroup$
    – rationalis
    Dec 30, 2020 at 1:12
  • $\begingroup$ @whuber I forgot the ||...||^2 in the first equation for the distribution - the Gaussian distribution exp(-(x_1^2 + x_2^2+...)). Since in C artesian coordinates the volume element is 1, both the "function" and the density have the same plot and are peaked at the center. No bubble, "No more than a bubble" by Jamel Brinkley. But if you write this in spherical coordinates and integrate over the D-1 angles (what the next commenter calls "projection"), only then do you get the apparent densification in the radial coordinate. $\endgroup$
    – ClimberT8
    Dec 31, 2020 at 3:03
  • $\begingroup$ @rationalis Sure, you are more than welcome to hold on to your preconception, but it is purely an artifact of projection. So you might as well ask "Why cubes are like hexagons?". And because you can't imagine or know of visualizations in higher dimensions please don't claim that they "simply" doesn't exist. $\endgroup$
    – ClimberT8
    Dec 31, 2020 at 3:08
  • $\begingroup$ @ClimberT8 There are many possible visualizations of higher dimensional space. None of them, however, are perfectly accurate. Perhaps a savant might be able to visualize a thousand dimensions, but I think it's pretty clear no human could visualize a billion dimensions. $\endgroup$
    – rationalis
    Jan 15, 2021 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.