1
$\begingroup$

I have often heard the argument that in higher dimensions: the "mode" (most common value) of a probability distribution function does not correspond to the "expectation" (mean) of the probability distribution function. The implication of this argument is that it becomes less meaningful to calculate the "mode" of a probability distribution function (i.e. the Bayesian MAP estimate), but rather it now becomes more advantageous to calculate the "expected value" of the probability distribution function (e.g. via MCMC).

I am trying to better understand this logic:

  1. I have heard that in high dimensional space, (it can be mathematically proven that) the majority of the volume of a hypersphere becomes concentrated around a thin outer ring of the hypersphere. If you apply this metaphor to high dimensional data, this means that the majority of the data is located around the extremities of the space, whereas most of the space itself is likely to be "sparsely populated". I can understand this argument. (e.g. Imagine an onion ring (thin torus) - the data would be the "onion crust" and all the volume in the middle is empty)

  2. What I don't understand is the following: In high dimensional space, why should the difference between the "mode" and the "expectation" of the probability distribution function be any more different than in low dimensional space? In high dimensional space, why is the "mode" not as useful (in terms of information, in terms of describing the probability distribution) as the "expectation"? In high dimensional space, logic tells us that most of the space is empty and the data is concentrated around the exterior. One could make the argument that "most of the data is located around the exterior", therefore the "mode" and "expectation" should be both as informative?

I have attached this picture as reference: https://imgur.com/a/pOj8FpP enter image description here Can someone please help me better understand this?

$\endgroup$
5
  • 5
    $\begingroup$ You don’t need high dimensions e.g. in 1D you just need an asymmetrical distribution, like exponential, for mode not to correspond to expected value. $\endgroup$
    – Tim
    Sep 20 '21 at 15:40
  • $\begingroup$ Does this answer your question? How best to summarize a predictive discrete distribution in a single number? $\endgroup$
    – Xi'an
    Sep 20 '21 at 15:49
  • 2
    $\begingroup$ @Tim One does not even need an asymmetrical distribution: $|x|\mathbb I_{[-1,1]}$ is a (bimodal) symmetric pdf with mean $0$ but modes $\pm 1$. $\endgroup$ Sep 20 '21 at 16:11
  • 1
    $\begingroup$ But the question is if this becomes more pronounced with increasing dimension ... $\endgroup$ Sep 20 '21 at 16:19
  • 4
    $\begingroup$ The "mode" displayed in Figure 9-4 is not the mode of the distribution. It is the mode of the distribution of radial distances from the mode of the distribution, as explained in the horizontal axis label. $\endgroup$
    – whuber
    Sep 20 '21 at 16:29
4
$\begingroup$

It is not clear what exactly the OP is asking. Consider, for example, the density of the vector random variable $(X_1, X_2, \ldots, X_n)$ whose component $X_i$ are $n$ independent standard normal random variables. The mode of this density (the location of the global maximum of the value of the density) is the origin. The expected value of the vector random variable is the vector of expected values of the individual random variables, and thus is also the origin. Neither of these parameters is a satisfactory measure of where the bulk of the probability mass lies: most of the probability mass lies in a (fairly large volumed) shell quite distant from the origin. Note that $R^2 = \sum_k X_k^2$ is a $\chi^2$ random variable with $n$ degrees of freedom. For large values of $n$, most of the probability mass of this $\chi^2$ random variable lies in the region $[n-\sqrt{18n} < R^2 < n+\sqrt{18n}]$. Put another way, if we look at $P\left(\sum_k X_k^2 < r^2\right)$ as a function of $r$, this quantity increases very very slowly when $r$ is small -- despite the density having large value there -- and then increases rapidly when $r$ is close to $\sqrt{n}$, arriving at a value quite close to $1$ -- this is the shell where most of the probability mass is -- and then once again returns to its very very slow rate of increase, asymtpotically approaching its final value of $1$. If we look at the graph in the OP's question, we see this quite clearly. For $n=10$, the peak is at $r \approx\sqrt{10} = 3.16\cdots$; for $n=100$ at $r \approx10$, and for $n=1000$, at $r\approx \sqrt{1000} = 10\sqrt{10} =31.6\cdots$.

If the question is, as Moderator Kjetil B Halvorsen says, whether the effect becomes more pronounced as $n$ gets larger, the answer is Yes, the distance of this peak from the mode or expected value increases with $n$ but only sublinearly. Fortunately for us, the curse of dimensionality does not increase as $O(n)$, only as $O(\sqrt n)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.