How to make sense out of integration over discrete data points?

Question

Looking for a proof of the expected value of the score function equating zero, I came to this document that was recommended in another answer.

Considering that we have a sample of $n$ $x_i$ values, I cannot figure out why the expected value turns out into an integral instead of a summation: what is the curve of which we're taking the area below it? In my mind I can just see some specific points in a graph, with no area under it, since we have a finite and discrete number of datapoints.

I do understand that the integral is crucial for the proof as to be interchanged with the derivative and then using the pdf probabilities for equating it to 1. But I would not know how to apply all this into a pmf or discrete case.

Thanks in advance

Answer 1

$X_i$ is continuous a random variable, with pdf $f_{X_i}(x_i;\theta)$, and the expectation requires an integral. The integral limits contain the domain of $X_i$. Not $i$ from $1$ to $n$. The $n$ samples you have are just realizations of $X_i$, i.e. $X_1,X_2,...,X_n$. You're not integrating/summing across these variables. You're integrating for a particular $i$, let's say $X_2$, and obtain an expression for the expected value of interest.

Answer 2

A full understanding of this issue requires a theory of integration over probability distributions, not just functions. However, even in such an abstract theory it's possible to visualize the integrals as areas under curves. The universal principle is that in any "reasonable" theory of integration, it should be possible to integrate by parts.

---

Consider the usual integral formulation of an expectation of a function $S$ for a distribution $F$ with density function $f(x) = F^\prime(x).$ This is given by

$$E_X[S(X)] = \int_{-\infty}^\infty S(x) f(x) \mathrm{d}x.$$

Let's suppose $S$ has two properties, neither of which severely limits the theory:

1. $S$ is differentiable and

2. The limiting values of $S(x)F(x)$ at $-\infty$ and $S(x)(1-F(x))$ at $\infty$ are zero. (This is equivalent to assuming $S$ has an expectation.)

The first enables us to apply integration by parts while the second enables us to cope with the infinite limits of integration. To do this, we will need to break the integral into two at some convenient (finite) value; for simplicity, let's break it at zero. In the negative region, write $f(x) = F^\prime(x)$ but in the positive region, $f(x) = -\frac{d}{dx}(1-F(x)).$ Integrating each integral separately by parts gives

$$\eqalign{ E_X[S(X)] &= &\int_{-\infty}^0 S(x) f(x) \mathrm{d}x + \int_0^\infty S(x) f(x) \mathrm{d}x \\ &= &\left(S(x)F(x)\left|_{-\infty}^0\right. - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x\right) + \\&&\left(-S(x)(1-F(x))\left|_0^\infty\right. + \int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x\right) \\ &= &\int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x.\tag{*} }$$

We may picture this process by drawing the areas under consideration, ignoring the factor of $S^\prime (x)$ for the moment:

The left image graphs the density function $f,$ the middle graphs the distribution function $F,$ and the right graphs the function $F$ for negative values of $x$ and $1-F$ for positive values. When you scale the heights of the right hand graph by the values of $S^\prime(x),$ the expectation is the corresponding (signed) area under the curve.

Turn now to a distribution without a density, such as a discrete distribution. Here are corresponding graphs for a distribution that puts probability $1-p$ on the value $-1$ and $p$ on the value $1$ (a Rademacher distribution):

(The plot of the density $f$ is omitted because, although it exists as a density, it does not exist as a function and therefore has no graph.)

---

As an example of how $(*)$ works, let's compute an expectation for this distribution. The integrals are finite because when $x \lt -1,$ $F(x)=0$ and when $x \ge 1,$ $1-F(x)=0.$ Thus:

$$\eqalign{ E[S] &= \int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x \\ &= \int_0^1 S^\prime(x)(1 - (1-p)) \mathrm{d}x - \int_{-1}^0 S^\prime(x) (1-p)\mathrm{d}x\\ &=(1 - (1-p))S(x)\left|_0^1\right. - (1-p) S(x)\left|_{-1}^0 \right. \\ &= (1-p)S(-1) + pS(1). }$$

This is the sum of the values of $S$ (at $\pm 1$) multiplied by their probabilities. A generalization of this calculation shows that this integral is precisely a sum of values multiplied by probabilities for any discrete distribution:

When $F$ is a discrete distribution supported at values $x_1,x_2,x_3, \ldots,$ with corresponding probabilities $p_1, p_2, p_3, \ldots,$ then the expression $(*)$ is $$E[S(X)] = \int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x = \sum_{i=1}^\infty S(x_i)p_i.$$ The integrals can be interpreted as signed areas, even though $F$ has no density function. Indeed, when $S^\prime$ is piecewise continuous, the integrals can be interpreted as Riemann integrals.

Answer 3

This proof corresponds to the case of a single data point (so $n=1$ in this context), where the distribution of the random variable $X_i$ is continuous, so it has a probability density function $f$. The proof uses the integral form from the law of the unconscious statistician, which holds that the expected value of the score function is an integral of that function multiplied by the density of $X_i$, taken over the full range of that random variable.

If $X_i$ were instead assumed to be a discrete random variable, instead of a continuous random variable, then the expected value would be a sum taken with respect to the mass function, instead of an integral taken with respect to the density function.

Answer 4

The proof you are examining starts by assuming $f(x_i; θ)$ is "a regular pdf." A pdf, or probability density function, is, by definition a continuous (i.e. not discrete) function. Since $X_i$ is continuous (hence pdf), you would use an integral to obtain the expected value of a function of $X_i$ by the Law of the Unconscious Statistician.