I understand the distinction between probability mass and density functions. But I don't understand what it means for a continuous random variable to have a probability distribution but not a density. I am reading this paper where the author consistently refers to the possibility that the probability density $P_{\theta}$ may or may not exist for a probability distribution $\mathbb{P}_{\theta}$. Can anyone explain what it meant by this? I would have thought that if $P_{\theta}$ did not exist, neither would $\mathbb{P}_{\theta}$.

  • $\begingroup$ The answer is prominently posted on the first page of the paper: "We need the model density $\mathbb{P}_\theta$ to exist. This is not the case in the rather common situation where we are dealing with distributions supported by low dimensional manifolds. It is then unlikely that the model manifold and the true distribution's support have a non-negligible intersection (see [1]), and this means that the KL distance is not defined (or simply infinite)." The distinction looks rather important to me--it certainly isn't "completely useless" (to quote a now deleted comment). $\endgroup$ – whuber Jun 22 '17 at 20:36
  • $\begingroup$ Just so we're on the same page, you have the notation in the paper wrong. He uses $\mathbb{P}_{\theta}$ to denote the distribution and $P_{\theta}$ to denote the density. Anyway, I do not understand that answer because the distinction in terms does not seem necessary to make his point. For example, why can't we also say that we need the model distribution to exist? And if the model density $P_{\theta}$ does not exist, how can $\mathbb{P}_{\theta}$ exist? $\endgroup$ – gwg Jun 22 '17 at 22:16
  • $\begingroup$ Sorry about not picking up on that typographical distinction--but isn't the quotation perfectly clear? Distributions always exist but densities need not exist. A good example is the Bernoulli$(p)$ distribution: its distribution function at $x$ equals $1-p$ when $0\le x\lt 1$ and otherwise is $0$ when $X\lt 0$ or $1$ when $X\ge 1$. It has no density. The paper is concerned about situations like a bivariate random variable $(X,Y)$ where $X$ has a standard Normal distribution and $Y=X$. This is perfectly well defined; it has a distribution function; but it cannot have a density. $\endgroup$ – whuber Jun 22 '17 at 22:21
  • $\begingroup$ In case you haven't encountered the definition of multivariate distributions, they are like the univariate one: $F(x,y) = \Pr(X\le x\text{ and }Y \le y)$. For instance, if $\Phi$ is the standard Normal distribution, then the distribution function for the $(X,Y)$ defined in my previous comment is $$F(x,y)=\Phi(\min(x,y)).$$The corresponding density function, if it exists, will equal $\frac{\partial^2}{\partial x\partial y}F$. However, this $F$ is not differentiable anywhere where $x=y$--the graph of $F$ has a sharp "ridge" there--and otherwise the mixed partials are zero. $\endgroup$ – whuber Jun 22 '17 at 22:27
  • $\begingroup$ Sorry if I am missing something obvious. I think I follow everything you've said so far. For example, the Bernoulli distribution does not have a density because a Bernoulli r.v. is discrete. But the paper seems to suggest there is something deeper here (maybe I am just reading into it): "The problem this paper is concerned with is that of unsupervised learning. Mainly, what does it mean to learn a probability distribution? The classical answer to this is to learn a probability density." Why a density? Why not just say distribution? $\endgroup$ – gwg Jun 22 '17 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.