32
$\begingroup$

According to the Page 87 in Kruschke: Doing Bayesian Data Analysis, the author says that the mean of a distribution is a value that minimizes the variance of a probability distribution, for example, a normal distribution. The following is what is mentioned in the page:

It turns out that the value of $ M $ that minimizes $ ∫dx p(x)(x−M)^2$ is $E[X] $. In other words, the mean of the distribution is the value that minimizes the expected squared deviation. In this way, the mean is a central tendency of the distribution.

I have read the paragraph and kind of understood what the author is trying to say but I wonder how this can be written mathematically using the above equation or any other ways.

$\endgroup$
3

7 Answers 7

58
$\begingroup$

Budding statisticians in Statistics 101 with no mathematical skills beyond high-school algebra should consider

\begin{align} E\left[(X-a)^2\right] &= E\bigr[\big(X-\mu + \mu -a\big)^2\bigr] & {\scriptstyle{\text{Here,}~\mu ~ \text{denotes the mean of} ~ X}}\\ &= E\bigr[\big((X-\mu) + (\mu -a)\big)^2\bigr]\\ &= E\bigr[(X-\mu)^2 + (\mu -a)^2 &{\scriptstyle{(\alpha+\beta)^2 = \alpha^2+\beta^2 + 2\beta\alpha}}\\ & ~~~~~~~~~~~~~~+ 2(\mu-a)(X-\mu)\bigr]\\ &= E\big[(X-\mu)^2\big] + E\big[(\mu -a)^2\big] &{\scriptstyle{\text{Expectation of sum is the sum of}}}\\ & ~~~~~~~~~+ 2(\mu-a)E\big[X-\mu\big] &{\scriptstyle{\text{the expectations, and constants}}}\\ & &{\scriptstyle{\text{can be pulled out ofexpectations}}}\\ &= \operatorname{var}(X) + (\mu -a)^2 + 2(\mu-a)\times 0 &{\scriptstyle{\text{definition of variance; expectation}}}\\ & &{\scriptstyle{\text{of a constant equals the constant;}}}\\ & &{\scriptstyle{E[X-\mu] = E[X] -E[\mu] = \mu -\mu = 0}}\\ &= \operatorname{var}(X) + (\mu -a)^2\\ &\geq \operatorname{var}(X) &{\scriptstyle{\text{with equality holding when}~ a=\mu.}} \end{align} Thus, we have shown that $E\left[(X-a)^2\right] \geq \operatorname{var}(X)$ with equality holding when $a = \mu$.

Look, Ma! No calculus! No derivatives or second derivative tests! Not even geometry and invocations of Pythogoras; just high-school algebra (even pre-algebra might have sufficed), together with a just a smidgen of Stats 101.

$\endgroup$
7
  • $\begingroup$ What do your equations say? $\endgroup$ Apr 18, 2021 at 16:33
  • 2
    $\begingroup$ @StoryMay My $E[(X-a)^2]$ is a succinct expression for what Kruschke's book (more likely, your transcription of it in your question) would have written as $ ∫dx p(x)(x−a)^2$, and has the advantages that (i) it applies to discrete as well as continuous random variables such as the normal random variables you mention, and (ii) it does not bring in that $> = E[X] $ that you (or Kruschke) seem to think is a lower bound on $ ∫dx p(x)(x−a)^2$. There is no requirement in mathematical statistics that the variance be lower-bounded by the mean; what machine learning thinks is a different matter. $\endgroup$ Apr 18, 2021 at 16:58
  • 2
    $\begingroup$ @StoryMay For example the Cauchy distribution has infinite variance, so the idea of "minimum variance" is meaningless. However any finite sample drawn from the Cauchy distribution (or other "long tailed" distributions) will have a finite variance - just plug the numbers into the formulas and get "the answer". Ignoring this inconvenient fact is the reason why much of econometrics (and other statistical forecasting) is garbage - the forecasts are "sort of OK" most of the time, except when they are catastrophically wrong. $\endgroup$
    – alephzero
    Apr 18, 2021 at 18:29
  • 2
    $\begingroup$ Excellent, accessible exposition! Although, I don't think you needed to say "no invocations of Pythagoras" because your answer helped me to understand Xi'an's. $\endgroup$
    – Neil G
    Apr 19, 2021 at 2:05
  • 2
    $\begingroup$ The useful result from this is $E\left[(X-a)^2\right] = \operatorname{var}(X) + (\mu -a)^2$. The conclusion of equality holding only when $a=\mu$ depends on $\operatorname{var}(X)$ being finite. $\endgroup$
    – Henry
    Apr 19, 2021 at 12:01
29
$\begingroup$

The expression is $\mathbb E[(X-a)^2]$. We'll differentiate and equate the expression to $0$: $$\begin{align}\frac{d}{da}\mathbb E[(X-a)^2]&=\mathbb E\left[\frac{d}{da}(X-a)^2\right]\\&=\mathbb E[-2(X-a)]\\&=0\end{align}$$

Then, $\mathbb E[-2X +2a]=0\rightarrow \mathbb E[2X]=\mathbb E[2a]=2a\rightarrow a=\mathbb E[X]$. The second derivative is positive, so it's the minimizer.

$\endgroup$
2
  • 3
    $\begingroup$ Thank you for the answer. Can you please show me the mathematical steps to your conclusion of $ a = E[X} $ or give some explanation why it is calculated that way? $\endgroup$ Apr 18, 2021 at 10:34
  • 5
    $\begingroup$ Thank you for updating the answer! $\endgroup$ Apr 18, 2021 at 10:46
7
$\begingroup$

Assuming that you have $n$ values $\{x_1, x_2, \ldots, x_n\}$, the mean squared difference from each value $x_i$ to some number $a$ is: $$ m(a)=\frac{1}{n}\sum_{i=1}^{n}(x_i-a)^2 $$ We could ignore the term $\frac{1}{n}$ but I'm leaving it in. Expand the square and manipulate: $$ m(a)=\frac{1}{n}\left(\sum_{i=1}^{n}x_{i}^{2}-2ax_i+a^2\right)=\frac{1}{n}\left(\sum_{i=1}^{n}x_{i}^{2}-\sum_{i=1}^{n}2ax_i+\sum_{i=1}^{n}a^2\right)=\frac{1}{n}\left(\sum_{i=1}^{n}x_{i}^{2}-2a\sum_{i=1}^{n}x_i+na^2\right) $$ To find the value of $a$ that minimizes the expression, differentiate with respect to $a$ to get: $$ m'(a)=\frac{1}{n}\left(2na - 2\sum_{i=1}^{n}x_i\right) $$ Now set this to zero an solve for $a$ to get: $$ a=\frac{\sum_{i=1}^{n}x_i}{n} $$ which is the mean of the values $x_i$. To show that this is indeed the minimum, calculate the second dervative wrt to $a$, which is $m''(a)=2>0$ so it's a minimum.

$\endgroup$
1
  • 2
    $\begingroup$ (+1) this is for the sample mean and the variance. $\endgroup$
    – gunes
    Apr 18, 2021 at 10:29
4
$\begingroup$

A quick one:

This follows from a property of moments (a rule to transform the center)

$$E\left[(x-\hat{x})^n\right] = \sum_{i=0}^n {n \choose i} E\left[(x-a)^i\right] (a-\hat{x})^{n-i}$$

which becomes for $n=2$ and $a=\mu=E[X]$

$$E \left[(x-\hat{x})^2\right] = \underbrace{E \left[(x-\mu)^2\right] }_{=\text{Var}(x)} + 2 \underbrace{E \left[(x-\mu)\right] }_{=0} (\mu-\hat{x}) + (\mu-\hat{x})^2 = \text{Var}(x) + (\mu-\hat{x})^2 $$

and this is minimised when $\hat{x}=\mu$.


A longer one:

It goes similar using the expression $∫dx p(x)(x−M)^2$

$$\begin{array}{} ∫dx p(x)(x−M)^2 &=& ∫dx p(x)\underbrace{((x-\mu)+(\mu-M))^2}_{=(x-\mu)^2 + 2(x-\mu)(\mu-M) + (\mu-M)^2} \\ & =& ∫dx p(x)(x-\mu)^2 + ∫dx p(x)2(x-\mu)(\mu-M) +∫dx p(x) (\mu-M)^2 \\ & =& ∫dx p(x)(x-\mu)^2 + ∫dx p(x)2 x(\mu-M) - ∫dx p(x)2\mu(\mu-M)+∫dx p(x) (\mu-M)^2 \\ & =& \underbrace{∫dx p(x)(x-\mu)^2}_{=\text{var}(x)} + 2 (\mu-M)\underbrace{∫dx p(x)x}_{=\mu} - 2 (\mu-M)\mu \underbrace{∫dx p(x)}_{=1}+(\mu-M)^2 \underbrace{∫dx p(x)}_{=1} \\ &=& \text{var}(x) +(\mu-M)^2 \end{array}$$


Intuitively:

In other words, the variance is the 2nd moment about the mean and a 2nd moment about some other point will be larger.

In order to find the optimum you could differentiate the integral and set equal to zero then you get

$$\frac{\partial}{\partial M} ∫dx p(x)(x−M)^2 =2 ∫dx p(x)(x−M) = 2(\mu-M) =0 $$

Which results in $M=\mu$.

In words: if you shift the point $M$ then the contributions of the $(x-M)^2$ term change and become less or more, and this is more when the distance $x-M$ is larger. When $M$ is equal to the mean $\mu$ the increase and decrease (the sum/mean of $x-M$) balance each other and you reach the minimum.

Something similar can be done to minimize the integral $∫dx p(x)|x−M|$ and you will find that this is minimised when $M$ is equal to the median.

$\endgroup$
4
$\begingroup$

Just work it out.

$$\frac{\partial}{\partial c}E[(X-c)^2]=E\left[ \frac{\partial}{\partial c}(X-c)^2 \right]=2E\left[ X-c \right]$$

Setting this to zero, $E[X]=c$ is the only stationary point and it's obviously not a maximum.

.

(there's an invocation of the dominated convergence theorem or near offer in there to justify putting the derivative through the integral, which might fail if the variance doesn't exist or something pathological like that)

$\endgroup$
3
  • $\begingroup$ Very elegant proof, love it! I was wandering in the wrong direction (chebyshev's inequality). Calculus is the way to go! $\endgroup$ Apr 25, 2021 at 22:27
  • 1
    $\begingroup$ I think we can differentiate $E[X^2]-2cE[X]+c^2$ with respect to $c$ without using the dominated convergence theorem, as long as we assume that $E[X^2]$ is finite (i.e. the variance exists). $\endgroup$
    – fblundun
    Apr 25, 2021 at 22:33
  • $\begingroup$ I wouldn't be at all surprised. I just couldn't be bothered working it out. ;) $\endgroup$ Apr 26, 2021 at 0:22
2
$\begingroup$

I have another geometric perspective which my college suggested me. In essence "the value $M$" that you are looking for is a factor such that $M\cdot\mathbb{1}$ is a projection of $X$ onto subspace generated by $\mathbb{1}$ in $L^2(\Omega, A, P)$ space where $\mathbb{1}$ is a constant random variable equal 1 almost everywhere.

Recall that if we have an $\mathbb{R}^n$ space with an inner product $\langle\cdot,\cdot\rangle$ and we want to project vector $p$ on subspace generated by vector $x$ then the projected vector is given by formula $$v=\frac{\langle p,x\rangle}{||x||^2}x$$ and $v$ it is a unique vector of form $Mx$ which minimizes $||p-Mx||^2.$

Instead of regular $\mathbb{R}^n$ consider space $L^2(\Omega, A, P)$ (space of random variables $X:\Omega\to\mathbb{R}$ such that $\operatorname{E}[X^2]<\infty$) with an inner product given by formula $$\langle X_1,X_2\rangle = \operatorname{E}[X_1\cdot X_2]=\int_\Omega X_1(x)X_2(x)dP(x).$$ If we fix an arbitrary random variable $X$ from $L^2(\Omega, A, P)$ and we want to project it onto subspace generated by random variable $\mathbb{1}$, then it is the same of finding $M$ such that it minimizes $$ \operatorname{E}[(X-M\cdot\mathbb{1})^2].$$ Using the same argument as in the $\mathbb{R}^n$, we get that $M=\frac{\langle X,\mathbb{1}\rangle}{||\mathbb{1}||^2}=\operatorname{E}[X]$.

$\endgroup$
1
$\begingroup$

$$ E[(X-M)^2]=E[X^2]-2ME[X]+M^2\Rightarrow M=E[X] $$ because minimum of quadradic polynomial $aM^2+bM+c$ is at $M=-b/(2a)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.