14
$\begingroup$

The question I want to ask is this: how does the proportion of samples within 1 SD of the mean of a normal distribution vary as the number of variates increases?

(Almost) everyone knows that in a 1 dimensional normal distribution, 68% of samples can be found within 1 standard deviation of the mean. What about in 2, 3, 4, ... dimensions? I know it gets less... but by how much (precisely)? It would be handy to have a table showing the figures for 1, 2, 3... 10 dimensions, as well as 1, 2, 3... 10 SDs. Can anyone point to such a table?

A little more context - I have a sensor which provides data on up to 128 channels. Each channel is subject to (independent) electrical noise. When I sense a calibration object, I can average a sufficient number of measurements and get a mean value across the 128 channels, along with 128 individual standard deviations.

BUT... when it comes to the individual instantaneous readings, the data does not respond so much like 128 individual readings as it does like a single reading of an (up to) 128-dimensonal vector quantity. Certainly this is the best way to treat the few critical readings we take (typically 4-6 of the 128).

I want to get a feel for what is "normal" variation and what is "outlier" in this vector space. I'm sure I've seen a table like the one I described that would apply to this kind of situation - can anyone point to one?

$\endgroup$
  • $\begingroup$ Please - can I have empirical answers only - I do not understand most mathematical notation. $\endgroup$ – omatai Dec 21 '11 at 0:46
19
$\begingroup$

Lets take $X = (X_1,\dots,X_d) \sim N(0,I)$ : each $X_i$ is normal $N(0,1)$ and the $X_i$ are independant – I guess that’s what you mean with higher dimensions.

You would say that $X$ is within 1 sd of the mean when $||X|| < 1$ (the distance between X and its mean value is lower than 1). Now $||X||^2 = X_1^2 +\cdots+X_d^2\sim \chi^2(d)$ so this happens with probability $P( \xi < 1 )$ where $\xi\sim\chi^2(d)$. You can find this in good chi square tables...

Here are a few values:

$$\begin{array}{ll} d& P(\xi < 1)\\ 1 & 0.68\\ 2 & 0.39 \\ 3 & 0.20 \\ 4 & 0.090 \\ 5 & 0.037 \\ 6 & 0.014 \\ 7 & 0.0052 \\ 8 & 0.0018\\ 9 & 0.00056\\ 10& 0.00017\\ \end{array}$$

And for 2 sd:

$$\begin{array}{ll} d & P(\xi < 4)\\ 1 & 0.95\\ 2 & 0.86\\ 3 & 0.74\\ 4 & 0.59\\ 5 & 0.45\\ 6 & 0.32\\ 7 & 0.22\\ 8 & 0.14\\ 9 & 0.089\\ 10 & 0.053\\ \end{array}$$

You can get these values in R with commads like pchisq(1,df=1:10), pchisq(4,df=1:10), etc.

Post Scriptum As cardinal pointed out in the comments, one can estimate the asymptotic behaviour of these probabilities. The CDF of a $\chi^2(d)$ variable is $$F_d(x) = P(d/2,x/2) = {\gamma(d/2, x/2) \over \Gamma(d/2)}$$ where $\gamma(s,y) = \int_0^y t^{s-1} e^{-t} \mathrm d t$ is the incomplete $\gamma$-function, and classicaly $\Gamma(s) = \int_0^\infty t^{s-1} e^{-t} \mathrm d t$.

When $s$ is an integer, repeated integration by parts shows that $$ P(s,y) = e^{-y} \sum_{k=s}^\infty {y^k \over k!}, $$ which is the tail of the CDF of the Poisson distribution.

Now this sum is dominated by its first term (many thanks to cardinal): $P(s,y) \sim {y^s \over s!} e^{-y}$ for big $s$. We can apply this when $d$ is even: $$P(\xi < x) = P(d/2,x/2) \sim {1 \over (d/2)!} \left({x\over 2}\right)^{d/2} e^{-x/2} \sim {1\over\sqrt{\pi d}}e^{{1\over 2}(d-x)} \left({x\over d}\right)^{d\over 2} \sim {1\over\sqrt\pi} e^{-{1\over 2}x} d^{-{1\over 2}d},$$ for big even $d$, the penultimate equivalence using Stirling formula. From this formula we see that the asymptotic decay is very fast as $d$ increase.

$\endgroup$
  • $\begingroup$ Welcome to our site, Elvis! Nice answer. (+1) $\endgroup$ – whuber Dec 21 '11 at 3:06
  • 1
    $\begingroup$ (+1) Good answer. Here are a couple suggestions for your consideration: (1) It might help to make explicit what $\xi$ is for clarity's sake, (2) briefly give an intuitive argument for the choice you've made for the meaning of "one standard deviation" in this context and why it is even well-defined in the first place, and (3) add a statement regarding the growth of this quantity as a function of $d$. (The OP asks for only "empirical" answers, but other readers might appreciate a small mathematical addendum.) $\endgroup$ – cardinal Dec 21 '11 at 4:22
  • $\begingroup$ Thank you for your comments. I didn’t think this answer would receive much attention! It is true that this is a nice form of the curse of dimensionality... @cardinal concerning (3) I don’t know any asymptotic equivalent of the incomplete gamma function when the first parameters goes to infinity, the second being fixed, this is not easy! A rough majoration could be done, I may write that later. $\endgroup$ – Elvis Dec 21 '11 at 8:31
  • 2
    $\begingroup$ Regarding (3), to avoid a computation, you can employ the following argument: Let $d$ be even and such that $d = 2 k$. Note that $Z_i = X_{2i-1}^2 + X_{2i}^2$ is an $\mathrm{Exp}(1/2)$ random variable. So $\|X\|^2 = \sum_{i=1}^k Z_i$. But, then $\|X\|^2$ is just the time until the $k$th renewal of a Poisson process with rate 1/2. So $\mathbb P(\|X\|^2 < 1 ) = \mathbb P( N_{1/2}(0,1) \geq k) = e^{-1/2} \sum_{x=k}^\infty 2^{-x}/x!$. The tail of the Poisson is dominated by the leading term, so $\mathbb P(\|X\|^2 < 1) \sim e^{-1/2} 2^{-k} / \Gamma(k+1)$ as $d\to\infty$ (Again: $k = d/2$). $\endgroup$ – cardinal Dec 21 '11 at 12:45
  • 1
    $\begingroup$ Part of the point of the foregoing comment is that we get an exact answer for all even $d$. Also, using Stirling's approximation, we get that $\mathbb P(\|X\|^2 < 1 ) \sim e^{-1/2} 2^{-k} / \Gamma(k+1) \sim e^{(d-1)/2} d^{-(d+1)/2} / \sqrt{\pi}$. $\endgroup$ – cardinal Dec 21 '11 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.