Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I was playing around in R and have gotten myself very confused about the relation between probability distributions, their expected values, and their cumulative distribution functions.

Say we're interested in the expected value of the gamma distribution. Here's a hacky way to calculate it in R:

library("pracma")
m=2
s=1
grid = seq(0,50,.01)
y = dgamma(grid, m, s)
(expected = trapz(grid,grid*y))
[1] 2

Intuitively, the expected value of the distribution is something like its 'center of mass': a point where half the weight of the distribution is on one side, and half is on the other side.

If this reasoning is right (I thought to myself), then this point should be identical to the point where the cumulative distribution function is equal to .50. Why? Well, because at the X value where the CDF is .5, a random variable has 50% change of being less than X and a 50% chance of being greater than X-- that is, X is the point where half the weight of the distribution lies on either side. (EDIT: If I had to pinpoint where my misconception occurred, I would say it was here. Center of mass isn't just about proportion of weight, as the answers below have reminded me.)

I decided to follow this hunch:

(halfway  = qgamma(.5, m, s))
[1] 1.678347

Nope, my hunch was wrong. The numerical difference seems to be too systematic to be the result of using numerical integration (seems to happen on any asymmetric distribution, to varying degrees, try for yourself).

How am I thinking about this incorrectly?

share|improve this question
1  
In physics a center of mass differs from what you describe: it is a "balancing point" of an object. As a little contemplation of mobiles, cantilevered architecture, and other highly asymmetric but stable objects will show, much more than half the mass can lie on one side of the center of mass ("barycenter" or "centroid"). The physical analog here is excellent, because probability density and mass density are mathematically identical. –  whuber Jun 10 at 13:55

3 Answers 3

up vote 6 down vote accepted

In the cases where it (the mean) exists, a nice way to look at this is that the mean is the value $M_2$ which minimizes $$ \int_{\text{support of $x$}} |x-M_2|^2 f(x) dx. $$ That is, the the distance from the mean is considered unevenly over the support.. all else equal the farther a point $x$ is from the mean the higher its 'cost' (if the distance is > 1).

The value for which $F(x) = 0.5$ is in fact the median, which is the value $M_1$ which minimizes $$ \int_{\text{support of $x$}} |x-M_1|^1 f(x) dx. $$ Distributions which are skewed* in one direction will have medians which are not in the same place as their means.. because the means will gravitate towards the longer tail to compensate for the squared distance, which penalizes further values more than the absolute difference does (when the distance is > 1).


*As rightly pointed out by @Glen_b skewness is used here as defined by Pearson's 2 skewness coeficient. That is, skew is $3\frac{\text{mean} - \text{mode}}{\text{std. deviation}}$.

share|improve this answer
1  
"which are skewed in one direction will have medians which are not in the same place as their means" unless you define skewness to make this true (define it as second Pearson skewness, say), this is not always the case. Usually, but not always. –  Glen_b Jun 10 at 2:30
    
Every real number minimizes the third integral, which equals $1$ no matter what value $M_0$ may have. –  whuber Jun 10 at 4:43
1  
I've added an explanation for the last equation, which I think will make clearer what Matthew means. –  Pat Jun 10 at 9:51
    
@Pat $0^0$ is always defined as $1$. See, for instance, mathforum.org/dr.math/faq/faq.number.to.0power.html. Regardless, the argument about "losing an infinitesimal amount of probability" is wrong. –  whuber Jun 10 at 13:12
2  
@whuber i removed the section on the mode since it wasn't even part of the question anyway –  Matthew Jun 10 at 13:48

Intuitively, the expected value of the distribution is something like its 'center of mass': a point where half the weight of the distribution is on one side, and half is on the other side.

This is correct, it is a center of mass. But beware about how you interpret that; you can't just say "half the mass is each side".

If this reasoning is right (I thought to myself), then this point should be identical to the point where the cumulative distribution function is equal to .50. Why? Well, because at the X value where the CDF is .5, a random variable has 50% change of being less than X and a 50% chance of being greater than X-- that is, X is the point where half the weight of the distribution lies on either side.

No, that's the median. They'll be the same when the distribution is symmetric (assuming the mean exists at all, naturally), but when it's non-symmetric, the two rarely* coincide.

* some people mistakenly assert 'never'. It's unusual, but possible

The moment has two components - the sign (which side you're on) and the magnitude - how far away you are. It's not sufficient to say there are the same fraction on either side**, so we must be at the mean - it depends on how far away they are.

**(or the same number of values on either side for a sample)

Consider a sample, like 1,2,18 (or a distribution with 1/3 probability at each point, it doesn't matter). Let's take "2" as being our candidate for the mean - after all, by your logic, there's the same proportion either side, so they should balance, right?

But the thing is, by putting a value 16 units above the proposed mean, it exerts 16 times the moment (in the physical sense) as the value one unit below it - just as a person sitting at the far end of a see-saw doesn't balance someone of the same weight sitting partway along the other arm:

enter image description here

So it's not having half the distribution either side that determine the mean.

However, if you weight those proportions by their distance you will get the balance you're thinking of (quite literally - take a very light, stiff rod and hang weights at 1, 2 and 18 distance units along the rod and find where they balance ... which will be at 7 units. If you work out the signed distances (-6,-5,+11) you can see they balance out.

If you check various distributions on Wikipedia, you'll find that typically for asymmetric distributions the mean and the median differ.

Indeed, the difference between them is used as the basis of a measure of skewness (with appropriate rescaling) - the second Pearson skewness measure.


Another note about your question: when you say "the expected value of $f(X)$", you misspeak. You intend to say "the expected value of $X$, where $X$ has density $f$". The expected value of a continuous $X$ is $\int x\,f(x) dx$. The expected value of $f(X)$, on the other hand is the expected value of the transformed random variable $Y=f(X)$, which by the law of the unconscious statistician is $\int f(x).f(x) dx$. That's a different object altogether. Just stick with "the expected value of $X$".

share|improve this answer
    
Thanks for the informative answer, and the terminological correction! –  jwdink Jun 10 at 2:25

$F(x)=0.5$ gives $x$ as the median of the distribution, which you could say is the centre of probability mass.

I think your understanding of the expected value being the 'centre of mass' of a distribution is a little misguided. Expected value is more of a weighted average than a centre of mass, in the discrete (and finite) case what you're doing is weighting the possible values your distribution can take on by the probability of that value occurring. In continuous distributions the analogy isn't as clear, but you're still weighting possible values your distribution can take on by their probability.

share|improve this answer
    
When a line is given a mass density proportional to a probability density, then indeed the mean is the center of mass: the analogy (between probability density and mass density) is perfect. –  whuber Jun 10 at 4:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.