Highest probability set and density ratios equal to probability ratios

Question

I came across a pretty result I had not seen before, and wondered if there were more examples

1. For a random variable with an exponential distribution, if you want the highest probability set to contain all but $p$ of the probability, then you want the density of the values you exclude to be less than $p$ times the density at the mode, and this is easy to show.

   • For example if $p=0.05$ and the exponential distribution has parameter $1$,
   • then you want to exclude values with $x > -\log_e\left(0.05\right) \approx 3$
   • and all their densities are below $0.05$, while the density at the mode at $0$ is $1$;
   • the excluded set covers $5\%$ of the probability. No surprises here.

2. For a bivariate normally distributed random variable, if you want the highest probability set to contain all but $p$ of the probability, then again you want the density of the values you exclude to be less than $p$ times the density at the mode. This is true even if the two parts of the bivariate distribution are correlated. This seems less obvious and is the pretty result I noticed; it can be proved as a consequence of the earlier exponential example and the relationship between the exponential distribution and the chi-squared distribution with $2$ degrees of freedom. This normal distribution result does not apply in $1$ dimension or in more than $2$ dimensions.

   • For example if $p=0.05$ and you have a standard uncorrelated bivariate normal with mean and mode $\mathbb \mu = {0 \choose 0}$ and covariance matrix $\mathbb \Sigma = {1 \: 0 \choose 0 \: 1}$,
   • then you want to exclude values where $\|\mathbf x\| > \sqrt{-2 \log_e\left(0.05\right)} \approx 2.45$
   • and all their densities are below $\frac{1}{40\pi}$, which is $0.05$ times the density at the mode of $\frac{1}{2\pi}$;
   • again the excluded set covers $5\%$ of the probability.

Are there other distributions with this property?

It would not particularly surprise me if there were artificial examples in higher dimensions, or simpler examples for single values of $p$, but are there any simple general examples like the two above?

Answer 1

I am not aware of any other distributions that have this property, though it would certainly be possible to construct distributions of that kind. Nevertheless, what I can offer is to give some structure to the problem by presenting the relevant functions. Following standard notation in the field, I will use $\alpha$ in place of your $p$, so that you are referring to a HDR with coverage probability $1-\alpha$.

---

Consider a continuous distribution with density function $f_X$ having supremum $m_X \equiv \sup_x f_X(x)$. We can define the function $H:[0,1] \rightarrow [0,1]$ via the equation:

$$1-\alpha = \mathbb{P}(f_X(X) \geqslant m_X \cdot H(\alpha)).$$

This does not uniquely define $H$, so it is usual to further stipulate that $H(\alpha)$ is the infimum of the set of all values for which the above equation is satisfied. If $f_X$ has no flat regions in its support then $H$ will also be continuous, but if there are flat regions in the support of the density the $H$ may have "jumps". Any any case, it is simple to show that $H$ is a non-decreasing function with $H(0) = 0$ and $H(1) = 1$.

For a HDR with coverage probability $1-\alpha$, suppose we let $r(\alpha)$ denote the cut-off point for the density (i.e., the HDR includes all points with $f_X(x) \geqslant r(\alpha)$). Then we can write our function of interest as:

$$H(\alpha) = \frac{r(\alpha)}{m_X}.$$

It is possible to derive this function for a number of families of distributions, though in some cases the function will only be implicitly defined. The condition of interest to you is when $H$ is the identity function (i.e., when $H(\alpha) = \alpha$). Below we will derive this function for various distributions, to show some distributions that have the property of interest to you, and some distributions that do not have this property.

---

The exponential distribution: Suppose that $f_X(x) = \text{Exp}(x|\theta)$ where $\theta$ is the rate parameter. Using the coverage probability $1-\alpha$ the HDR and density cut-off are:

$$\text{HDR}(1-\alpha) = \Bigg[ 0, \frac{|\ln(\alpha)|}{\theta} \Bigg] \quad \quad \quad r(\alpha) = \theta \cdot \alpha.$$

Since $m_X = \theta$ we therefore have:

$$H(\alpha) = \frac{r(\alpha)}{m_X} = \frac{\theta \cdot \alpha}{\theta} = \alpha.$$

This confirms your observation that the exponential distribution has the property of interest. As you have pointed out in the comments, it also applies to the chi-squared distribution with two degrees-of-freedom, since that is an exponential distribution with $\theta=\tfrac{1}{2}$.

---

The normal distribution: Suppose that$f_X(x) = \text{N}(x|\mu, \sigma^2)$ where $\mu$ is the mean parameter and $\sigma$ is the standard deviation parameter. Using the coverage probability $1-\alpha$ the HDR and density cut-off are:

$$\text{HDR}(1-\alpha) = \Bigg[ \mu - z_{\alpha/2} \sigma, \ \mu + z_{\alpha/2} \sigma \Bigg] \quad \quad \quad r(\alpha) = \frac{1}{\sqrt{2 \pi \sigma^2}} \cdot \exp \bigg( -\frac{z_{\alpha/2}^2}{2} \bigg).$$

Since $m_X = 1/\sqrt{2 \pi \sigma^2}$ we therefore have:

$$H(\alpha) = \frac{r(\alpha)}{m_X} = \exp \bigg( -\frac{z_{\alpha/2}^2}{2} \bigg).$$

---