Probability Distributions : "Mode" vs. "Expectation"

Question

I have often heard the argument that in higher dimensions: the "mode" (most common value) of a probability distribution function does not correspond to the "expectation" (mean) of the probability distribution function. The implication of this argument is that it becomes less meaningful to calculate the "mode" of a probability distribution function (i.e. the Bayesian MAP estimate), but rather it now becomes more advantageous to calculate the "expected value" of the probability distribution function (e.g. via MCMC).

I am trying to better understand this logic:

1. I have heard that in high dimensional space, (it can be mathematically proven that) the majority of the volume of a hypersphere becomes concentrated around a thin outer ring of the hypersphere. If you apply this metaphor to high dimensional data, this means that the majority of the data is located around the extremities of the space, whereas most of the space itself is likely to be "sparsely populated". I can understand this argument. (e.g. Imagine an onion ring (thin torus) - the data would be the "onion crust" and all the volume in the middle is empty)

2. What I don't understand is the following: In high dimensional space, why should the difference between the "mode" and the "expectation" of the probability distribution function be any more different than in low dimensional space? In high dimensional space, why is the "mode" not as useful (in terms of information, in terms of describing the probability distribution) as the "expectation"? In high dimensional space, logic tells us that most of the space is empty and the data is concentrated around the exterior. One could make the argument that "most of the data is located around the exterior", therefore the "mode" and "expectation" should be both as informative?

I have attached this picture as reference: https://i.stack.imgur.com/fOGE6.jpg Can someone please help me better understand this?

Answer 1

It is not clear what exactly the OP is asking. Consider, for example, the density of the vector random variable $(X_1, X_2, \ldots, X_n)$ whose component $X_i$ are $n$ independent standard normal random variables. The mode of this density (the location of the global maximum of the value of the density) is the origin. The expected value of the vector random variable is the vector of expected values of the individual random variables, and thus is also the origin. Neither of these parameters is a satisfactory measure of where the bulk of the probability mass lies: most of the probability mass lies in a (fairly large volumed) shell quite distant from the origin. Note that $R^2 = \sum_k X_k^2$ is a $\chi^2$ random variable with $n$ degrees of freedom. For large values of $n$, most of the probability mass of this $\chi^2$ random variable lies in the region $[n-\sqrt{18n} < R^2 < n+\sqrt{18n}]$. Put another way, if we look at $P\left(\sum_k X_k^2 < r^2\right)$ as a function of $r$, this quantity increases very very slowly when $r$ is small -- despite the density having large value there -- and then increases rapidly when $r$ is close to $\sqrt{n}$, arriving at a value quite close to $1$ -- this is the shell where most of the probability mass is -- and then once again returns to its very very slow rate of increase, asymtpotically approaching its final value of $1$. If we look at the graph in the OP's question, we see this quite clearly. For $n=10$, the peak is at $r \approx\sqrt{10} = 3.16\cdots$; for $n=100$ at $r \approx10$, and for $n=1000$, at $r\approx \sqrt{1000} = 10\sqrt{10} =31.6\cdots$.

If the question is, as Moderator Kjetil B Halvorsen says, whether the effect becomes more pronounced as $n$ gets larger, the answer is Yes, the distance of this peak from the mode or expected value increases with $n$ but only sublinearly. Fortunately for us, the curse of dimensionality does not increase as $O(n)$, only as $O(\sqrt n)$.