Looking for a proof of the expected value of the score function equating zero, I came to this document that was recommended in another answer.

Proof of the expected value of the score function

Considering that we have a sample of $n$ $x_i$ values, I cannot figure out why the expected value turns out into an integral instead of a summation: what is the curve of which we're taking the area below it? In my mind I can just see some specific points in a graph, with no area under it, since we have a finite and discrete number of datapoints.

I do understand that the integral is crucial for the proof as to be interchanged with the derivative and then using the pdf probabilities for equating it to 1. But I would not know how to apply all this into a pmf or discrete case.

Thanks in advance

  • $\begingroup$ This is the score for a single observation $x_i$. An integral is used because $X_i$ is a continuous random variable. $\endgroup$ – Ahmed Ali Jun 17 '19 at 0:38

$X_i$ is continuous a random variable, with pdf $f_{X_i}(x_i;\theta)$, and the expectation requires an integral. The integral limits contain the domain of $X_i$. Not $i$ from $1$ to $n$. The $n$ samples you have are just realizations of $X_i$, i.e. $X_1,X_2,...,X_n$. You're not integrating/summing across these variables. You're integrating for a particular $i$, let's say $X_2$, and obtain an expression for the expected value of interest.

  • $\begingroup$ In that sense, if I understand correctly, we are actually computing the expected value of the score for one observation of the sample. Which is then generalized to the whole sample as they are all assumed to be i.i.d.? $\endgroup$ – Kuku Jun 18 '19 at 17:30
  • 1
    $\begingroup$ Yes, expectation is calculated for one $X_i$, and then generalized. $\endgroup$ – gunes Jun 19 '19 at 0:33

A full understanding of this issue requires a theory of integration over probability distributions, not just functions. However, even in such an abstract theory it's possible to visualize the integrals as areas under curves. The universal principle is that in any "reasonable" theory of integration, it should be possible to integrate by parts.

Consider the usual integral formulation of an expectation of a function $S$ for a distribution $F$ with density function $f(x) = F^\prime(x).$ This is given by

$$E_X[S(X)] = \int_{-\infty}^\infty S(x) f(x) \mathrm{d}x.$$

Let's suppose $S$ has two properties, neither of which severely limits the theory:

  1. $S$ is differentiable and

  2. The limiting values of $S(x)F(x)$ at $-\infty$ and $S(x)(1-F(x))$ at $\infty$ are zero. (This is equivalent to assuming $S$ has an expectation.)

The first enables us to apply integration by parts while the second enables us to cope with the infinite limits of integration. To do this, we will need to break the integral into two at some convenient (finite) value; for simplicity, let's break it at zero. In the negative region, write $f(x) = F^\prime(x)$ but in the positive region, $f(x) = -\frac{d}{dx}(1-F(x)).$ Integrating each integral separately by parts gives

$$\eqalign{ E_X[S(X)] &= &\int_{-\infty}^0 S(x) f(x) \mathrm{d}x + \int_0^\infty S(x) f(x) \mathrm{d}x \\ &= &\left(S(x)F(x)\left|_{-\infty}^0\right. - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x\right) + \\&&\left(-S(x)(1-F(x))\left|_0^\infty\right. + \int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x\right) \\ &= &\int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x.\tag{*} }$$

We may picture this process by drawing the areas under consideration, ignoring the factor of $S^\prime (x)$ for the moment:


The left image graphs the density function $f,$ the middle graphs the distribution function $F,$ and the right graphs the function $F$ for negative values of $x$ and $1-F$ for positive values. When you scale the heights of the right hand graph by the values of $S^\prime(x),$ the expectation is the corresponding (signed) area under the curve.

Turn now to a distribution without a density, such as a discrete distribution. Here are corresponding graphs for a distribution that puts probability $1-p$ on the value $-1$ and $p$ on the value $1$ (a Rademacher distribution):

Figure 2

(The plot of the density $f$ is omitted because, although it exists as a density, it does not exist as a function and therefore has no graph.)

As an example of how $(*)$ works, let's compute an expectation for this distribution. The integrals are finite because when $x \lt -1,$ $F(x)=0$ and when $x \ge 1,$ $1-F(x)=0.$ Thus:

$$\eqalign{ E[S] &= \int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x \\ &= \int_0^1 S^\prime(x)(1 - (1-p)) \mathrm{d}x - \int_{-1}^0 S^\prime(x) (1-p)\mathrm{d}x\\ &=(1 - (1-p))S(x)\left|_0^1\right. - (1-p) S(x)\left|_{-1}^0 \right. \\ &= (1-p)S(-1) + pS(1). }$$

This is the sum of the values of $S$ (at $\pm 1$) multiplied by their probabilities. A generalization of this calculation shows that this integral is precisely a sum of values multiplied by probabilities for any discrete distribution:

When $F$ is a discrete distribution supported at values $x_1,x_2,x_3, \ldots,$ with corresponding probabilities $p_1, p_2, p_3, \ldots,$ then the expression $(*)$ is $$E[S(X)] = \int_0^{\infty} S^\prime(x) (1-F(x)) \mathrm{d}x - \int_{-\infty}^0 S^\prime(x) F(x) \mathrm{d}x = \sum_{i=1}^\infty S(x_i)p_i.$$ The integrals can be interpreted as signed areas, even though $F$ has no density function. Indeed, when $S^\prime$ is piecewise continuous, the integrals can be interpreted as Riemann integrals.

  • $\begingroup$ I'm afraid your answer leaves me with more questions than at the beginning. I'll try to go from more general to more specific: (a) Is it standard nomenclature to refer to a distribution by its CDF? (b) We have not characterized S(X) or F(X) in any way, aren't we assuming here by doing the split at 0 in the way it was defined that the pdf is symmetrical at 0? (c) For the same reason as in the previous point, in the explanation of (*), why are we assuming that F(0) = 1, for instance? (d) What do you mean by the sentence that f "exists as a density" but not as a function? Thanks in advance $\endgroup$ – Kuku Jun 18 '19 at 18:33
  • $\begingroup$ (a) Yes. (b) No: symmetry is not assumed. (c) I do not assume $F(0)=1.$ (d) A density is a mathematical object to assign numbers to sets, whereas a function assigns numbers to numbers. A "density function" $f,$ aka pdf, achieves both purposes by assigning the number $\int_{\mathcal A}f(x)\mathrm{d}x$ to any set $\mathcal A$ where that integral makes sense. However, not all densities can be so represented. For instance, the density that assigns a value $p$ to any set containing $1$, $1-p$ to any set containing $-1,$ and $1$ to any set containing both $1$ and $-1$ has no density function. $\endgroup$ – whuber Jun 18 '19 at 18:46

This proof corresponds to the case of a single data point (so $n=1$ in this context), where the distribution of the random variable $X_i$ is continuous, so it has a probability density function $f$. The proof uses the integral form from the law of the unconscious statistician, which holds that the expected value of the score function is an integral of that function multiplied by the density of $X_i$, taken over the full range of that random variable.

If $X_i$ were instead assumed to be a discrete random variable, instead of a continuous random variable, then the expected value would be a sum taken with respect to the mass function, instead of an integral taken with respect to the density function.

  • $\begingroup$ I feel your answer is in the same direction as the one from gunes, so I will repeat my comment here: if I understand correctly, we are actually computing the expected value of the score for one observation of the sample, which is then generalized to the whole sample as they are all assumed to be i.i.d.? $\endgroup$ – Kuku Jun 18 '19 at 18:34
  • $\begingroup$ Yes, that is correct - under IID condition the score function is the sum of the scores for the individual data points, so it suffices to look at the case of a single data point. $\endgroup$ – Ben Jun 19 '19 at 0:20
  • $\begingroup$ If $X_i$ is a discrete random variable, then the expected value is still the integral w.r.t to the mass function (counting measure) right? $\endgroup$ – LE Anh Dung Jan 16 '20 at 23:19
  • $\begingroup$ Yes, essentially. The general Lebesgue-Stieltjes would be the easiest way to write it (i.e., w.r.t. the distribution function), and then you don't need to worry if it is continuous or discrete, or a mixture. $\endgroup$ – Ben Jan 17 '20 at 0:27

The proof you are examining starts by assuming $f(x_i; θ)$ is "a regular pdf." A pdf, or probability density function, is, by definition a continuous (i.e. not discrete) function. Since $X_i$ is continuous (hence pdf), you would use an integral to obtain the expected value of a function of $X_i$ by the Law of the Unconscious Statistician.

  • $\begingroup$ If $X_i$ is a discrete random variable, then the expected value is still the integral (in the sense of counting measure) w.r.t to the mass function right? $\endgroup$ – LE Anh Dung Jan 16 '20 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.