5
$\begingroup$

I'm reading through class notes explaining statistical functionals and came across the following expressions with little explanation how they were derived:

Mean $=T(F)=\int xdF(x)$

Variance $=T(F)=\int (x-\mu)^2dF(x)$

Am I correct that $dF(x)=f(x)dx$? If so I'm not sure what the motivation is - what advantages do we gain expressing descriptive statistics in functional form?

$\endgroup$
0
9
$\begingroup$

$dF(x) = f(x)dx$ is true when the probability measure of the random variable (or equivalently the CDF, $F(x)$) is absolutely continuous with respect to the Lebesgue Measure. What that means is that for all sets with Lebesgue measure zero, the measure also assigns measure 0. You can think of this as meaning that the Lebesgue measure and the distribution measure are measures of the same space. In this case, we can define a "derivative" of the distribution of $X$ with respect to the Lebesgue measure:

\begin{align} \frac{dF(x)}{dx} = f(x) \end{align}

where $f(x)$ is the probability density function. We can also think of this as a change of measure, a reweighting of the probability distribution measure and the Lebesque measure. In general this is an example of a radon-nikodym derivative.

So why introduce all of this machinery and notation to get back to a pdf?

The real magic is in the case that $F(x)$ is not absolutely continuous with the Lebesgue measure. For example, say we have a discrete random variable. Discrete random variables cannot have probability measures absolutely continuous with Lebesgue because they explicitly give positive measure to some countable set.

In the discrete case, we take the radon-nikodym with respect to the counting measure and the integrals in this case simplify as following:

\begin{align} E[g(X)] = \int g(x)dF(x) = \sum_{x\in \mathcal{X}} g(x)P(X=x) \end{align}

where $\mathcal{X}$ is the support of the discrete random variable $X$. In elementary statistics and probability books often discrete and continuous random variables are presented completely separately. Essentially we define the radon-nikodym derivative with respect to the counting measure, give it a special name (pmf) then talk about how we can construct integrals of interest with respect to these pmfs. Then we start all over defining the radon-nikodym derivative for continuous variables (pdfs) and reintroduce all the same concepts with respect to pdfs.

The measure theory framework allows us to generalize the concept to include both continuous and discrete random variables (as well as those which don't fit into either category) and introduce a standard notation that does not force us to make distinctions which are unimportant with respect to the underlying probability theory, fostering connections between concepts. The cost is obviously an investment in measure theory, which may not be feasible for all people interested in learning stats and probability.

$\endgroup$
5
  • 1
    $\begingroup$ Thank you Tyrel, this is helpful. I should mention the class notes I'm reading are discussing statistical functionals as foundational to understanding influence functions. So it looks like we're building towards being able to take a derivative of a nonparametric estimator (like a sample mean or sample variance) that isn't normally differentiable, is that correct? Or even a machine learning algorithm like a neural network in the case of targeted maximum likelihood estimation (TMLE). $\endgroup$
    – RobertF
    Oct 19 '20 at 13:46
  • 1
    $\begingroup$ It's not so much about taking the derivatives of the models themselves, but as @Xi'an say below about expressing these complicated estimators of functions in terms of the empirical distribution and harnessing the functional convergence of the empirical process. Although in your notes you can see in the formula's for influence functions that it is useful to have a notion of derivative that does not depend directly on the type of distribution. $\endgroup$ Oct 19 '20 at 16:22
  • $\begingroup$ Ok, but how can you take an integral of a discrete random variable, vs. a summation? Does the Radon-Nikodym Theorem find a way to integrate over point masses or does it simply state "let's make it possible 'cause it allows us to do other useful mathematics". $\endgroup$
    – RobertF
    Oct 19 '20 at 17:34
  • 1
    $\begingroup$ In the answer I posted, I show the example of a discrete random variable, but I did it for some general expectation $E[g(X)]$ (most any integral of interest in stats can be expressed as an expectation).. Maybe to be more concrete, let's take the mean. Suppose $X$ is a discrete random variable. In your notation $E[X] = T(F) = \int xdF(x) = \sum_{x\in\mathcal{X}}xP(X=x)$. The integral is a sum. Sums are just a special type of integral really and the radon-nikdym theory gives us a way of seeing that. Is that more clear? $\endgroup$ Oct 19 '20 at 17:48
  • $\begingroup$ It looks similar to a Riemann sum where you can partition an area/volume under a curve into rectangles with width shrinking to zero at the limit? $\endgroup$
    – RobertF
    Oct 20 '20 at 1:31
6
$\begingroup$

First a correction to the question: when writing $$T_1(F) = \int x\text d F \qquad \text{or} \qquad T_2(F) = \int (x-T_1(F))^2\,\text d F$$ these quantities are not statistics but functionals of the distribution of the data. For instance, $T_1(F)=\mathbb E [X]$ is the mean of $F$. Functionals are thus generalised moments. (Statistics are coming into my second point, below.) As for the use of $\text D F$, this is a generic notation in measure theory and Tyrel Stokes's answer is excellent.

Concerning statistics, the motivation for the notation is at the core of the bootstrap approach: when estimating $T(F)$, one uses instead $T(\hat F_n)$ when $\hat F_n$ is the empirical distribution, based on the observed sample. Since this empirical distribution has a finite support, it does not enjoy a pdf wrt Lebesgue measure but has a pmf over the set defined by the observed sample. The reasoning is that, since $\hat F_n$ is a convergent approximation of $F$ (by the Glivenko-Cantelli theorem), the same should apply to $T(\hat F_n)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.