I have the following set of continuous measurements:


To average this data, I sum the values and divide by the number of values to get 197.884. This suggests to some degree that the probability of each is 1/n = 1/9, and we are getting this average by the following calculation:

$$E(X)=\sum_{i=1}^{9}{p_{i}\cdot x_{i}} = \sum_{i=1}^{9}{\frac{1}{9}\cdot x_{i}}$$

...where $p_{i}$ is the probability of each (1/9) and $x_{i}$ is the random value associated with each probability.

This approach for summing is generally reserved for discrete data, but this is continuous data. In reality, $p_{i}$ for each value above is not an even 1/9 across the board, but follows some unknown probability density curve $f(x)$.

Should “$f(x)$” be known, then I could easily use the expectation operator to find the average:

$$E(X) = \int_{x=a}^{b}{x\cdot f\left( x \right)}dx$$

However, in this data set, the $f(x)$ is unknown, so this approach can’t be done.

Obviously we must find the average by summing these values and dividing by “n” as shown above, but this treats the values as discrete data, and not as continuous data.

In essence, I am curious how in these two approaches, $p_{i} = \frac{1}{9}$ is bridged to the unknown $f(x)$ for the continuous data?

Clearly if $f(x)$ follows some known function's behavior, then we can fit $f(x)$ to the measured points and then have ourselves a continuous probability density function to use with the expectation operator, but I’m not seeing how the value found by discrete averaging is similar to that of the expectation operator used on the (currently unknown) probability density function?

From what I can tell, all descriptive calculations done on this data would have to treat it as discrete points, and not as continuous values, even though in reality they are continuous measures. This relation between the continuous and discrete handling of this data does not make sense to me, and its what I’m dying to understand.

  • 1
    $\begingroup$ You seem to be conflating a sample mean with a population expectation. $\endgroup$
    – whuber
    May 1 '15 at 20:38

It seems like you think that, when we calculate the sample mean of a distribution, it's like we're trying to calculate the expected value of the random variable but using a discrete approximation to the actual distribution. That's not what we're doing, though, or at least not always the most fruitful way to think about it.

What we're actually doing is using a statistical estimator of the expected value. Now it happens that, in the case of estimating the expected value, the estimator we usually use has the same form as if we used a discrete approximation to the continuous distribution. But that's not always true.

For instance, when estimating the variance of a distribution, we sometimes don't do the naive thing and calculate the variance on the obvious discrete approximation to the distribution, because that's biased low. To get an unbiased estimate, we multiply the result of that procedure by $\frac{n}{n-1}$. If you try to view this as "doing the variance calculation on a discrete random variable", the discrete variable it ends up matching to is the obvious one times $\sqrt{\frac{n}{n-1}}$.

I don't think viewing the unbiased variance estimator as the exact variance of a certain discrete random variable sheds very much light on this. Rather, the calculation that you do to estimate the mean or variance or other statistics of a random variable should be viewed as motivated by the statistical properties you want the estimator to have (e.g. low bias, low sensitivity to outliers, low variance, etc.). Often it turns out that the calculation looks somewhat like calculating the equivalent statistic on a discrete random variable, but the causality goes "want statistical properties of an estimator => choose a calculation to run => interpret calculation as a statistic of a discrete random variable," not the other way around.

EDIT: As @guy pointed out in the comments, there actually is some rigorous motivation for why calculating the same statistic on the discrete random variable is often the right thing to do. As I understand it, the intuition is essentially the same as how I described a related issue in my answer to your previous question--as you get more samples, the distribution "pick a value from my sample at random" starts to look similar to "pick a value from the true distribution" under finer and finer quantizations. So this motivates taking the statistic on the discrete distribution as at least asymptotically reasonable. Of course, that raises the question of why the "pick a value from my sample at random" distribution is a good approximation, but I'm not sure I know how to explain that in a way that will dispel your confusion.

  • 1
    $\begingroup$ "That's not what we're doing, though, or at least not the most fruitful way to think about it." - I think there are a lot of people working in, and applying, the theory of empirical processes who would very much disagree with this statement :) Thinking in terms of $\mathbb P_n = \frac 1 n \sum_{i = 1} ^ n \delta_{X_i}$ is incredibly powerful. $\endgroup$
    – guy
    May 1 '15 at 22:48
  • $\begingroup$ @guy: hmm. Is there a better answer to this question based on empirical process theory? I'd love to hear it. $\endgroup$
    – Ben Kuhn
    May 2 '15 at 0:25
  • $\begingroup$ Well, one could argue that we are doing exactly what you say we aren't: constructing a discrete approximation to the continuous CDF and plugging the estimate into the functional of interest $\mu: P \mapsto \int x \ P(dx)$. This is often the optimal thing to do, at least asymptotically. I'm only claiming this is a fruitful way of thinking because it points us towards the statistical theory that powers many methods - not necessarily that it is the best answer to OPs question. $\endgroup$
    – guy
    May 2 '15 at 2:43
  • $\begingroup$ On a side note: it's somewhat interesting that you point to the plug-in estimate of the variance as being inferior considering (1) it is asymptotically equivalent to the bias-corrected estimate you gave and (2) it is actually uniformly better in finite samples under squared error loss! Even better is the best invariant estimator which takes $n+1$ in the denominator. I only mention this as a curiosity, since your point is valid in general that a plug-in estimator can be beat in finite samples. $\endgroup$
    – guy
    May 2 '15 at 2:46
  • $\begingroup$ With respect to "there is no particular reason this should be true", if the data is iid from $P$, then $\mathbb P_n$ is the NPMLE of $P$ (and there are Glivenko-Cantelli type results of course for uniformity of convergence) so one might expect by the invariance principle of MLEs in parametric families that, similarly, for any sufficiently regular functional $\nu$ that $\nu(\mathbb P_n)$ is asymptotically efficient as an estimator of $\nu(P)$ in the nonparametric family. $\endgroup$
    – guy
    May 2 '15 at 3:03

Let's imagine that we have the true probability density function $f(x)$. What would be the expectation of an average of 9 random values from this distribution?

$E\left[\frac{1}{9}\sum_{i=1}^9 x_i\right]=\frac{1}{9}\sum_{i=1}^9 E[x_i]=\frac{1}{9}\sum_{i=1}^9 \int_{-\infty}^\infty x_i\cdot f(x_i) dx_i$

The integral is the same for all 9 random variables: $\mu$ the expectation of the distribution. Hence, that sum is: $E\left[\frac{1}{9}\sum_{i=1}^9 x_i\right]=\frac{1}{9}\sum_{i=1}^9\mu=\frac{9\mu}{9}=\mu$

This means that the expectation of the simple average of 9 random numbers is the expectation of the distribution. In statistics this is called an estimator, and the notation is usually $\hat\mu=\frac{1}{n}\sum_{i=1}^n x_i$. It's a "good" estimator, because it converges in some sense to a true value.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.