How is typical set defined for general high-dimensional distributions? I'm only aware of the definitions in Elements of Information Theory, which deal with iid and stationary ergodic processes. From there we can speak of the typical set of, e.g., a high-dimensional standard normal distribution. But I've seen people referring to the "typical set" of general high-dimensional distributions (such as in Betancourt’s introduction to Hamiltonian Monte Carlo). Is there any rigorous definition in such cases?
 A: My understanding is that the rigorous definition of the $\delta$-typical set of $p$, relative to the base measure $p_0$, is given by
$$S_\delta ( p; p_0 ) = \left\{ x : \left| \log \frac{p (x)}{p_0 (x)} - \mathbf{E}_{p(\tilde{x})} \left[ \log \frac{p (\tilde{x})}{p_0 (\tilde{x})} \right]  \right| < \delta \right\}.$$
I cannot provide a reference for this, unfortunately. 
Note that the inclusion of the base measure is often left implicit, which can lead to confusion, as without it, the typical set will generally not be reparametrisation-equivariant.
A: I agree with @r8. We can see from e.g., stan or these notes that the weakly $\epsilon$-typical set is given by
$$
A_\epsilon = \{ x: \left|\ln \frac{p(x)}{\pi(x)} - H(p, \pi)\right| \le \epsilon \}
$$
where $\pi(x)$ is a reference measure and 
$$
H(p, \pi) = \int p(x) \ln  \frac{p(x)}{\pi(x)} dx
$$
Let me add that we may quickly demonstrate an inequality for the volume of the typical set. We begin by expressing the condition above as
$$
e^{H - \epsilon}\, \pi(x) \le p(x) \le e^{H + \epsilon}\, \pi(x)
$$
We then build an inequality
$$
1 \ge  \int_{A_\epsilon} p(x) dx \ge \int_{A_\epsilon} \pi(x) dx \, e^{H - \epsilon}
$$
and thus
$$
\text{Volume of typical set} \equiv \int_{A_\epsilon} \pi(x) dx \le e^{-H + \epsilon}
$$
Let me further add that in the context of inference from a prior to a posterior (which I why I chose the notation $\pi$ and $p$) we may write the condition for the typical set in terms of the likelihood function $\mathcal{L(x)}$,
$$
e^{H - \epsilon}\, \mathcal{Z} \le \mathcal{L(x)} \le e^{H + \epsilon}\, \mathcal{Z}
$$
where $\mathcal{Z}$ is the Bayesian evidence (also known as the marginal likelihood),
$$
\mathcal{Z} \equiv \int \mathcal{L}(x) \pi(x) dx
$$
