14
$\begingroup$

I understand the distinction between probability mass and density functions. But I don't understand what it means for a continuous random variable to have a probability distribution but not a density. I am reading this paper where the author consistently refers to the possibility that the probability density $P_{\theta}$ may or may not exist for a probability distribution $\mathbb{P}_{\theta}$. Can anyone explain what it meant by this? I would have thought that if $P_{\theta}$ did not exist, neither would $\mathbb{P}_{\theta}$.

$\endgroup$
10
  • 4
    $\begingroup$ The answer is prominently posted on the first page of the paper: "We need the model density $\mathbb{P}_\theta$ to exist. This is not the case in the rather common situation where we are dealing with distributions supported by low dimensional manifolds. It is then unlikely that the model manifold and the true distribution's support have a non-negligible intersection (see [1]), and this means that the KL distance is not defined (or simply infinite)." The distinction looks rather important to me--it certainly isn't "completely useless" (to quote a now deleted comment). $\endgroup$
    – whuber
    Jun 22, 2017 at 20:36
  • 3
    $\begingroup$ Sorry about not picking up on that typographical distinction--but isn't the quotation perfectly clear? Distributions always exist but densities need not exist. A good example is the Bernoulli$(p)$ distribution: its distribution function at $x$ equals $1-p$ when $0\le x\lt 1$ and otherwise is $0$ when $X\lt 0$ or $1$ when $X\ge 1$. It has no density. The paper is concerned about situations like a bivariate random variable $(X,Y)$ where $X$ has a standard Normal distribution and $Y=X$. This is perfectly well defined; it has a distribution function; but it cannot have a density. $\endgroup$
    – whuber
    Jun 22, 2017 at 22:21
  • 5
    $\begingroup$ In case you haven't encountered the definition of multivariate distributions, they are like the univariate one: $F(x,y) = \Pr(X\le x\text{ and }Y \le y)$. For instance, if $\Phi$ is the standard Normal distribution, then the distribution function for the $(X,Y)$ defined in my previous comment is $$F(x,y)=\Phi(\min(x,y)).$$The corresponding density function, if it exists, will equal $\frac{\partial^2}{\partial x\partial y}F$. However, this $F$ is not differentiable anywhere where $x=y$--the graph of $F$ has a sharp "ridge" there--and otherwise the mixed partials are zero. $\endgroup$
    – whuber
    Jun 22, 2017 at 22:27
  • 3
    $\begingroup$ The problem is that the simple classification into "discrete" and "continuous" no longer holds in higher dimensions. The example I gave is of a distribution that is singular in the sense it has no density (because it is supported on a set of Lebesgue measure zero) yet in a well-defined sense really is a continuous distribution when restricted to its support. As to why that paper insists on densities, I could not answer that without studying the paper. I would take it as an announcement that the authors have restricted their interests to problems that can be modeled with densities. $\endgroup$
    – whuber
    Jun 23, 2017 at 13:24
  • 1
    $\begingroup$ I am not aware of any extended analysis of distributions into "discrete" and "continuous" in higher dimensions. The situation is too complicated to admit of any simple (or, I suspect, useful) analysis. The principal distinctions are between "has a density," "is discrete" (that is, with point support), and "neither." Often the latter can be handled by approximating the distribution through adding a multivariate normal variable with very, very tiny variances: this is guaranteed to produce a distribution with a smooth density. $\endgroup$
    – whuber
    Jun 23, 2017 at 14:06

2 Answers 2

13
+50
$\begingroup$

I would suggest to the OP to read Koopmans, L. H. "Teaching singular distributions to undergraduates." The American Statistician 37, no. 4a (1983): 313-316.

My general impression is that "CDF without densities" arise in a way that can be "real-world intuitive" in multivariate settings rather than univariate. As he writes (p. 314)

Finally, whereas singular distributions are difficult (if not impossible) to visualize in one dimension, being distributions concentrated on uncountable zero- dimensional sets (so to speak), in two dimensions they are, or can be constructed to be, concentrated on rather familiar one-dimensional figures such as lines and circles.

He also provides a real-world example indeed (Example 1, p. 314).

Elaborating in a naive way on the bivariate CDF with which he starts, he considers

$$F(x,y) = \frac{x+y}{2},\;\;\;\; 0\leq x\leq 1,\;\; 0\leq y\leq 1.$$

Don't be misled by the notation that "points" towards "continuous distributions". After all this is exactly the issue here.

One can verify that the cross-partial of $F(x,y)$ is zero, $$\frac{\partial^2 F(x,y)}{\partial x \partial y} =0.$$

Strictly speaking the mathematical operation of computing the derivative appears legal, given the information we have, but by giving us the constant zero-function, it leaves us scratching our heads (Koopmans writes "At this point in an exam, it is not unreasonable to expect the normal undergraduate to panic.")

An indication that something peculiar may be going on here is by deriving the marginal CDF, say, for $X$, $$\Pr (X\leq x) = G(x) = \lim_{y\to 1}F(x,y) = \frac{x}{2} + \frac 12.$$ But this means that $\Pr (X=0) = \frac 12$, implying that $X$ is a random variable of "mixed" type (not mixture), and that its marginal CDF $G(x)$ has one continuous and a non-zero discrete component: $$G(x) = \begin{cases} 1/2\qquad\;\;\;\;\;\;\; x=0 \\ 1/2 + x/2 \;\;\;\; 0 < x \leq 1.\end{cases}$$

Koopmans shows that such a mixed distribution leads to a non-zero "singular" component, in the Lebesgue decomposition of a joint density in three parts, one continuous, one discrete and one singular.

$\endgroup$
1
2
$\begingroup$

As an alternative to Alecos' excellent answer, I will try to give a more intuitive explanation, with the help of two examples.

If the distribution of a random quantity $X$ has a density $\varphi$, numeric values for the probability that $X$ takes values in a given set $A$ can be obtained by integrating $\varphi$ over $A$: $$ P(X\in A) = \int_A \varphi(x) \, dx. $$

If $X$ takes real numbers as values, a consequence of this property is that the probability of $X$ taking any specific value equals zero: $$ P(X = a) = \int_{\{a\}} \varphi(x) \, dx = \int_a^a \varphi(x) \, dx = 0. $$ The following example shows a random quantity which is continuous but has $P(X=0) > 0$ and therefore cannot have a density.

Example 1. Let $X$ be the amount of rainfall in millimetres on this day next year. Clearly the amount of rainfall is a continuous quantity. But where I live there are only 120 days of rain per year, so it seems reasonable to assume $P(X=0) = (365-120)/365 \approx 2/3 > 0$. If we try to find a density for $X$ we may come up with something like the following picture:

an attempt to sketch a density of X

The problem is what we should do for $x=0$, where all the days with no rain are represented by a single point on the $x$-axis. To get a density, we would need a "peak" with width 0 and a height such that the integral over the peak equals $P(X=0)$. One might be tempted to set $\phi(0) = \infty$, but even if we accept such a $\phi$ as a valid density, it seems impossible to somehow encode the value $P(X=0) = 2/3$ in the function $\varphi$. Because of these complications, the distribution of $X$ cannot be represented by a density. The distribution can instead be described using a probability measure or the cumulative distribution function.

The same problem also appears in higher dimensions. If $X$ is a random vector with a density, then we still have $P(X=a) = 0$. Similarly, if $A$ is a set with area/volume zero, then the integral discussed above is zero and thus we also must have $P(X\in A) = 0$.

Example 2. Assume that $X = (X_1, X_2)$ is uniformly distributed on the unit circle in the Euclidean plane:

the unit circle in the Euclidean plane

Clearly this is a continuous distribution (there are uncountably many points in the unit circle), but since the circle line has area $0$ it does not contribute to the integral, and any function $\varphi$ with $\varphi(x) = 0$ outside the circle will have $$ \int_{-\infty}^\infty \int_{-\infty}^\infty \varphi(x_1, x_2) \, dx_2 \, dx_1 = 0. $$ Thus there cannot be a density function (integrating to 1) which only takes values on the circle and the distribution of $X$ cannot be described by a density on the Euclidean plane. It can instead be described by a probability measure or by using polar coordinates and using a density just for the angle.

$\endgroup$
3
  • 2
    $\begingroup$ Thank you for your thoughtful contributions. However, the senses in which you use "continuous" are not that of the "continuous random variable" in the question. A continuous random variable is one with a continuous distribution function, which is not the case with either of your examples. The first is a mixture of an absolutely continuous and discrete distribution while the second is a singular bivariate distribution. $\endgroup$
    – whuber
    Aug 18, 2022 at 16:24
  • $\begingroup$ @whuber ah yes, I used "continuous" in the sense described at en.wikipedia.org/wiki/Continuous_or_discrete_variable , i.e. the opposite of "discrete". It is not clear to me which sense of "continuous" the OP intends. My second example is inspired by the sentence "... situation where we are dealing with distributions supported by low dimensional manifolds ..." in the introduction of the paper the OP links to. $\endgroup$
    – jochen
    Aug 19, 2022 at 9:04
  • 2
    $\begingroup$ Continuous is not quite the opposite of "discrete." Lebesgue's Decomposition Theorem shows any measure is a mixture of three types: discrete, absolutely continuous, and "singular." The sense of "continuous" is clearly implied by the context. $\endgroup$
    – whuber
    Aug 19, 2022 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.