3
$\begingroup$

This has probably been asked before, as this is (I think) a fundamental theory of statistical theory, but I don't know what it is called, hence I have not yet found an answer.

Consider a box which produces random numbers.

The observed values are $x_i$.

If we have a total of $N$ observations, we could add them together. $\sum_{i=1}^N x_i$.

If we divide by $N$ we get something more interesting. $\frac{\sum x_i}{N}$ This is the mean.

If we were to assume there exists some underlying probability distribution $P(x)$ then we should be able to calculate the mean of this probability distribution. This is also the first moment of the distribution. It is obtained by integrating.

$$\int_x x\cdot P(x) \mathrm{d}x=\mu$$

How can we prove the two functions are related? There is a discrete form, and a continuous form. It should be possible to relate the first form to the second by taking a limit of $N\to\infty$. But I cannot rigerously convince myself how to connect the two. This seems like a difficult problem - but I am not sure if that is because it is actually trivial, or because this is genuinely difficult to prove.

How can we prove this?

$$\frac{\sum x_i}{N} \to \int_x x\cdot P(x) \mathrm{d}x=\mu$$

$\endgroup$
1
  • 3
    $\begingroup$ Search your textbooks or the Web (or even this site) for the "Weak Law of Large Numbers" and the "Strong Law of Large Numbers." They trace their origin to J. Bernoulli's posthumous Ars Conjectandi, published in 1713. $\endgroup$
    – whuber
    Aug 2, 2023 at 21:43

1 Answer 1

1
$\begingroup$

I think I have an answer to this now, however I am not exactly confident as to how mathematically rigerous this is.

Firstly let me address the Weak Law of Large Numbers.

The Weak Law does apply here, because it makes the following statement about sampling a large number of IID Random Variables from a distribution. It says that:

$$\lim_{n\to\infty}\sum_{i=1}^{n}\frac{X_i}{n}=\mu$$

where $\mu$ is obtained by integrating the underlying probability density function.

  • The Weak Law makes a statement about how far from the mean a set of $n$ samples is likely to be, in other words it relates probabilities to this sum.

However, what I had intended in asking my question was a bit of a different approach.

I think the answer is actually fairly trivial.

We make the following definition of the expectation value:

$$\mathbb{E}[{X_n}]\hat{=}\sum_i^n \frac{x_i}{n}$$

What can we say about the $x_i$? If we assume that these are drawn from an underlying probability density function, then we could calculate the weighted average.

At this point, I am not sure about the how rigerous this is. Is this a circular proof? I think not, and here is why:

We could imagine a specific probability density function $p(y)$. Each $x_i$ could take one of $m$ possible values of $y_j$, where $j\in[1,m]$.

At each value $p(y_j)$, we have a sampled value of the pdf $p_j$.

At this point I had drawn a diagram on paper with an arbitrary squiggly line for the pdf, and drawn evently spaced sampling points along the horrisontal axis. (Which confusingly here I have labeled as "$y_j$".)

We can imagine comparing this "sampled" pdf with different values of $m$. (Different numbers of samples.) If we sample it, and add up all the samples, then if we were to take more samples, the number we would end up with would be larger.

The pdf is everywhere $>=0$. If we sample it an infinite number of times, we obtain something which is infinite.

The solution is to normalize. Irrespective of how many samples of the pdf we take, we want the sum of those samples to be 1.

We are taking $m$ samples of $p(y_j)$. These are evenly spaced samples between the minimum and maximum values of $y_j$.

This is what we obtain, the weighted average of any $x_i$:

$$\frac{\sum_j p(y_j)y_j}{\sum_j p(y_j)}$$

When calculating this I made a number of intermediate steps, but I do not think these necessarily make the result any clearer. I will append those to the end of this question.

To continue, since all the $x_i$ are the same (iid), we obtain this:

$$\mathbb{E}[{X_n}]=\sum_i^n \frac{1}{n}\frac{\sum_j^m p(y_j)y_j}{\sum_j^m p(y_j)}$$

This seems to be quite a trivial result, and there isn't anything complicated about it. It is not quite the same as "just taking a weighted average", because although this is a weighted average, the more important point is that:

  • If $n$ is large, then although the individual values of $x_i$ are "fixed" values, and may be highly variable in value, we can move from a manner of thinking where $x_i$ is a number, to a manner of thinking where $x_i$ takes all possible values, but those values are weighted by their respective probabilities.

  • The respective probabilities are proportional to the underlying pdf, and the constant of proportionality is obtained by performing a normalization. Although the pdf itself is normalized, sampling it as uniformly spaced points is not. The more samples we take, the larger the resulting weighted value $x_i$ will be.

The set of $y_j$ and $p_j$ are discrete. We can obtain something continuous by multiplying by one and taking the limit $\lim m\to\infty$.

$$\lim_{m\to\infty}\sum_i^n \frac{1}{n}\frac{\sum_j^m p(y_j)y_j \frac{\Delta y}{m}}{\sum_j^m p(y_j)\frac{\Delta y}{m}}$$

$\Delta y = y_{max}-y_{min}$. $y_{max}$ and $y_{min}$ are arbitrarily large, and we can take them to infinity.

We obtain:

$$\sum_i^n \frac{1}{n}\frac{\int_y p(y_j)y_j \mathrm{d}y}{\int_y p(y_j)\mathrm{d}y}$$

The definition of the average of the underlying pdf, which is the first moment, exists in the above equation, and is $\mu$.

$$\sum_i^n \frac{1}{n}\frac{\mu}{1}=n\frac{1}{n}\mu=\mu$$

Therefore

$$\mathbb{E}[X_n]=\mu$$


Sampling the PDF:

$x_i$ is a idd random variable drawn from some pdf $p(y)$. We drop the index $i$.

$$x=\sum_j^m p_j y_j$$

where $p_j$ is the probability to obtain $y_j$. $p_j$ depends on how we sample the underlying pdf. It depends on $m$. We sample from some minimum and maximum value of $y$, $y_{max}$, $y_{min}$.

Because $p_j$ depends on $m$, better to write $p_m(y_j)$.

$$x=\sum_j^m p_m(y_j) y_j$$

$p_m$ is proportional to the pdf $p(y)$. Therefore $p_m(y_j)=\alpha_m p(m_j)$

The symbols $p(y_j)$ and $p_m(y_j)$ mean totally different things here. The former is the underlying pdf, and the latter is a sampled version of it.

From here, things are relatively trivial. We calculate $\alpha_m$ by normalizing:

$$1=\sum_j^m \alpha_m p(y_j)$$

hence

$$\alpha_m^{-1}=\sum_j^m p(y_j)$$


Not totally sure about this answer so some feedback would be helpful?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.