12
$\begingroup$

I am doing ML at my university, and the professor mentioned the term Expectation (E), while he was trying to explain some things on Gaussian processes to us. But from the way he explained it, I understood that E is the same as the mean μ. Did I understand right?

If it is the same, then do you know why both symbols are used? Also I saw that E can be used as a function, like E($x^2$), but I didn't see that for μ.

Can someone help me understand the difference between the two better?

Improve this question
$\endgroup$
5
  • $\begingroup$ For continuous $X$, $E[X] = \int_{-\infty}^{\infty} f(x)x dx = \mu(x)$ where $f(x)$ is the probability density function. So it is true only when $X$ is the argument. However it could also be true if we have $E[g(X)] = E[X] = \mu(X)$, where $g$ is something other than the identity function. $\endgroup$
    – Jase
    Feb 6, 2014 at 15:24
  • 2
    $\begingroup$ @Jase $\mu(x)$? Why is the right side a function of $x$, which should have disappeared after substitution of the limits while evaluating the integral? $\endgroup$
    – Dilip Sarwate
    Feb 6, 2014 at 15:26
  • 1
    $\begingroup$ @DilipSarwate $\mu(x)$ was a typo. Mean to say $\mu = \mu(X)$. $\endgroup$
    – Jase
    Feb 6, 2014 at 15:28
  • 3
    $\begingroup$ John: if I were you, I would learn basic probability before taking Machine Learning / Gaussian Processes classes. Take a look at this book: math.uiuc.edu/~r-ash/BPT.html $\endgroup$
    – Zen
    Feb 6, 2014 at 16:39
  • $\begingroup$ Thanks a lot guys for your help! I didn't expect so much feedback. @Zen Thanks a lot for your advice. I absolutely agree with you. I have taken a module as undergrad at probabilities and statistics, However, we just had a simple introduction at distributions, and probabilities, and unfortunately we didn't do them in depth. In addition we did not mentioned the term "Expectation". I am trying now, to cover my gaps at statistics and probabilities by myself. $\endgroup$
    – Jim Blum
    Feb 6, 2014 at 20:35

2 Answers 2

Reset to default
11
$\begingroup$

Expectation/Expected value is an operator that can be applied to a random variable. For discrete random variables (like binomial) with $k$ possible values it is defined as $\sum_i^k x_i p(x_i)$. That is, it's the mean of the possible values weighted by the probability of those values. Continuous random variables can be thought of as the generalization of this: $\int x dP$. The mean of a random variable is a synonym for expectation.

The Gaussian (normal) distribution has two parameters $\mu$ and $\sigma^2$. If $X$ is normally distributed, then $E(X)=\mu$. So the mean of a Gaussian distributed variable is equal to the parameter $\mu$.This is not always the case. Take the binomial distribution, which has parameters $n$ and $p$. If $X$ is binomially distributed, then $E(X)=np$.

As you saw, you can also apply expectation to functions of random variables so that for a gaussian $X$ you can find that $E(X^2)=\sigma^2+\mu^2$.

The Wikipedia page on expected values is pretty informative: http://en.wikipedia.org/wiki/Expected_value

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ "... so that for a Gaussian $X$ you can find that $E(X^2)=\sigma^2+\mu^2$." Is it absolutely necessary that $X$ by Gaussian for this relationship to hold? $\endgroup$
    – Dilip Sarwate
    Feb 6, 2014 at 15:32
  • $\begingroup$ The relationship $E(X^2)=V(X)+E(X)^2$ will always hold, but I would expect the answer written in terms of the parameters of the distribution. So if I asked someone what $E(X^2)$ was for $X$ distributed Binomial$(n,p)$, I would expect the answer $np(1-p)+(np)^2$, not $\sigma^2+\mu^2$ $\endgroup$
    – Jeremy Coyle
    Feb 6, 2014 at 17:26
  • $\begingroup$ But if you asked what was $E(X^2)$ for a binomial random variable with mean $\mu$ and variance $\sigma^2$, the answer would be $\sigma^2+\mu^2$. Granted that binomial random variables are usually parameterized using $n$ and $p$, but so what? From the mean and variance we can readily find $$p = 1 - \frac{\text{variance}}{\text{mean}}$$ and $$n = \frac{\text{mean}}{p} = \frac{\text{mean}^2}{\text{mean}-\text{variance}}.$$ $\endgroup$
    – Dilip Sarwate
    Feb 6, 2014 at 19:23
  • 1
    $\begingroup$ The whole point of the example was to make a distinction between parameters of a distribution and the moments of a distribution. Yes it is possible to reparameterize distributions in terms of their moments, but since the OP was asking about the relationship between $E(X)$ and $\mu$, it seems important to continue making that distinction. Is there a reason you're choosing to be pedantic about this point? $\endgroup$
    – Jeremy Coyle
    Feb 6, 2014 at 19:53
  • 1
    $\begingroup$ Thanks a lot Jeremy! Excellent answer. you were very helpful! $\endgroup$
    – Jim Blum
    Feb 6, 2014 at 20:36
8
$\begingroup$

Expectation with an operator notation $E()$ (varying preferences on good fonts, roman or italic, plain or fancy, are found) does imply taking the mean of its argument, but in a mathematical or theoretical context. The term goes back to Christiaan Huygens in the 17th century. The idea is explicit in much of probability theory and mathematical statistics and, for example, Peter Whittle's book Probability via expectation makes clear how it could be made even more central.

It is basically just a matter of convention that means (averages) are also often expressed rather differently, notably by single symbols, and especially when those means are to be calculated from data. However, Whittle in the book just cited uses a notation $A()$ for averaging and angle brackets around variables or expressions to be averaged are common in physical science.

See e.g. https://en.wikipedia.org/wiki/Peter_Whittle_(mathematician) on Whittle (1927-2021). The book in question went through two titles, three publishers and four English editions between 1970 and 2000, although the second edition is really a reprint. Although not elementary, it includes many different examples and emphases and sufficient wry comments to make it interesting and useful even to readers without a strong mathematical background (such as myself).

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.