I am doing ML at my university, and the professor mentioned the term Expectation (E), while he was trying to explain us some things on Gaussian processes. But from the way he explained it, I understood that E is the same as the mean μ. Did I understood right?

If it is the same, then do you know why both symbols are used? Also I saw that E can be used as a function, like E($x^2$), but I didn't see that for μ.

Can someone help me understand better the difference between the two?

  • $\begingroup$ For continuous $X$, $E[X] = \int_{-\infty}^{\infty} f(x)x dx = \mu(x)$ where $f(x)$ is the probability density function. So it is true only when $X$ is the argument. However it could also be true if we have $E[g(X)] = E[X] = \mu(X)$, where $g$ is something other than the identity function. $\endgroup$ – Jase Feb 6 '14 at 15:24
  • 1
    $\begingroup$ @Jase $\mu(x)$? Why is the right side a function of $x$, which should have disappeared after substitution of the limits while evaluating the integral? $\endgroup$ – Dilip Sarwate Feb 6 '14 at 15:26
  • 1
    $\begingroup$ @DilipSarwate $\mu(x)$ was a typo. Mean to say $\mu = \mu(X)$. $\endgroup$ – Jase Feb 6 '14 at 15:28
  • 2
    $\begingroup$ John: if I were you, I would learn basic probability before taking Machine Learning / Gaussian Processes classes. Take a look at this book: math.uiuc.edu/~r-ash/BPT.html $\endgroup$ – Zen Feb 6 '14 at 16:39
  • $\begingroup$ Thanks a lot guys for your help! I didn't expect so much feedback. @Zen Thanks a lot for your advice. I absolutely agree with you. I have taken a module as undergrad at probabilities and statistics, However, we just had a simple introduction at distributions, and probabilities, and unfortunately we didn't do them in depth. In addition we did not mentioned the term "Expectation". I am trying now, to cover my gaps at statistics and probabilities by myself. $\endgroup$ – Jim Blum Feb 6 '14 at 20:35

Expectation/Expected value is an operator that can be applied to a random variable. For discrete random variables (like binomial) with $k$ possible values it is defined as $\sum_i^k x_i p(x_i)$. That is, it's the mean of the possible values weighted by the probability of those values. Continuous random variables can be thought of as the generalization of this: $\int x dP$. The mean of a random variable is a synonym for expectation.

The Gaussian (normal) distribution has two parameters $\mu$ and $\sigma^2$. If $X$ is normally distributed, then $E(X)=\mu$. So the mean of a Gaussian distributed variable is equal to the parameter $\mu$.This is not always the case. Take the binomial distribution, which has parameters $n$ and $p$. If $X$ is binomially distributed, then $E(X)=np$.

As you saw, you can also apply expectation to functions of random variables so that for a gaussian $X$ you can find that $E(X^2)=\sigma^2+\mu^2$.

The Wikipedia page on expected values is pretty informative: http://en.wikipedia.org/wiki/Expected_value

  • 1
    $\begingroup$ "... so that for a Gaussian $X$ you can find that $E(X^2)=\sigma^2+\mu^2$." Is it absolutely necessary that $X$ by Gaussian for this relationship to hold? $\endgroup$ – Dilip Sarwate Feb 6 '14 at 15:32
  • $\begingroup$ The relationship $E(X^2)=V(X)+E(X)^2$ will always hold, but I would expect the answer written in terms of the parameters of the distribution. So if I asked someone what $E(X^2)$ was for $X$ distributed Binomial$(n,p)$, I would expect the answer $np(1-p)+(np)^2$, not $\sigma^2+\mu^2$ $\endgroup$ – Jeremy Coyle Feb 6 '14 at 17:26
  • $\begingroup$ But if you asked what was $E(X^2)$ for a binomial random variable with mean $\mu$ and variance $\sigma^2$, the answer would be $\sigma^2+\mu^2$. Granted that binomial random variables are usually parameterized using $n$ and $p$, but so what? From the mean and variance we can readily find $$p = 1 - \frac{\text{variance}}{\text{mean}}$$ and $$n = \frac{\text{mean}}{p} = \frac{\text{mean}^2}{\text{mean}-\text{variance}}.$$ $\endgroup$ – Dilip Sarwate Feb 6 '14 at 19:23
  • 1
    $\begingroup$ The whole point of the example was to make a distinction between parameters of a distribution and the moments of a distribution. Yes it is possible to reparameterize distributions in terms of their moments, but since the OP was asking about the relationship between $E(X)$ and $\mu$, it seems important to continue making that distinction. Is there a reason you're choosing to be pedantic about this point? $\endgroup$ – Jeremy Coyle Feb 6 '14 at 19:53
  • 1
    $\begingroup$ Thanks a lot Jeremy! Excellent answer. you were very helpful! $\endgroup$ – Jim Blum Feb 6 '14 at 20:36

Expectation with an operator notation E() (varying preferences on good fonts, roman or italic, plain or fancy, are found) does imply taking the mean of its argument, but in a mathematical or theoretical context. The term goes back to Christiaan Huygens in the 17th century. The idea is explicit in much of probability theory and mathematical statistics and, for example, Peter Whittle's book Probability via expectation makes clear how it could be made even more central.

It is basically just a matter of convention that means (averages) are also often expressed rather differently, notably by single symbols, and especially when those means are to be calculated from data. However, Whittle in the book just cited uses a notation A() for averaging and angle brackets around variables or expressions to be averaged are common in physical science.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.