Suppose $Y \sim Poisson(k \theta)$ is observed (number of events), where $k>0$ is a constant (length of observation time) and $\theta$ is an unknown parameter (event rate) with some prior distribution.

Is it possible to show that the expectation (with respect to $Y$ and $\theta$) of the Shannon entropy of the posterior is convex in $k$? i.e. the expected information gain is subject to decreasing returns in the length of observation time.

I'm interested in results for any particular choice of prior (except a point mass) or for all priors.


Answer based on the original question.

The Shannon entropy of the Poisson distribution is

$$\lambda [1-\log(\lambda )]+e^{-\lambda }\sum _{k=0}^{\infty }{\frac {\lambda ^{k}\log(k!)}{k!}},$$ where $\lambda = \theta k$ in your case. For each value of $\theta$, this seems to be a concave function of $k$:

# Approximation to the Shannon Entropy truncating the series at N
shan.pois <- Vectorize(function(k){
  x <- 1:N
  val <- (theta*k)*(1-log(theta*k)) + sum(  dpois(x, lambda = theta*k)*lgamma(x+1) )

# Plot for different values of theta
N <- 1000

theta <- 1
curve(shan.pois,0,10, n = 1000, ylim = c(0,5),lwd=2, xlab = "k", ylab = "S(k)")

thetas <- seq(0.1,10,by = 0.1)
for(i in 1:length(thetas)){
  theta <- thetas[i]
  curve(shan.pois,0,10, n = 1000, add = T, col = "grey")

So, even if you find a strongly informative prior that, for a specific sample, the posterior expectation is convex, it will eventually become concave by the concentration properties of the posterior distribution.

enter image description here

  • 1
    $\begingroup$ Thanks for the answer! Unless I'm missing a connection, I think you're looking at expectation of the entropy of y wrt the posterior distribution, but I'm interested in expectation of the entropy of the posterior distribution with respect to y and $\theta$. I'll reword the question to make it clearer. $\endgroup$ – Dennis Prangle Apr 24 '20 at 14:08
  • $\begingroup$ @DennisPrangle That is certainly a very different question! I will have a go. $\endgroup$ – Typo Apr 24 '20 at 14:10

Information theory representation

The expectation with respect to $Y$ of the posterior entropy is the conditional entropy $$ H(\theta | Y) = H(\theta) - \mathrm{I}(Y; \theta) $$ where $H(\theta)$ is prior entropy and $I$ is mutual information.

Let $g(k)$ denote the mutual information as a function of $k$. So the question can be rephrased as showing that $g$ is concave.

Submodularity of mutual information (discrete case)

The proof extends a similar result on mutual information under repeated observations.

Let $\mathcal{S} = \{ Z_1, Z_2,\ldots, Z_n \}$, a set of conditionally independent (given $\theta$) random variables with the same distribution. Define $f(\mathcal{A}) = \mathrm{I}(\mathcal{A}; \theta)$ where $\mathcal{A} \subseteq \mathcal{S}$.

Proposition 2 of Krause and Guestrin shows that $f(A)$ is a submodular function (the proof follows from submodularity of entropy).

(Edit: in fact a modification of this proposition is needed to deal with the fact that $\theta$ is a continuous parameter. See the proof of equation (25) in Appendix G of Prangle, Harbisher and Gillespie.)

That is, for every $\mathcal{A} \subseteq \mathcal{B} \subseteq \mathcal{S}$ and $Z_i \in \mathcal{S} \setminus \mathcal{B}$: $$f(\mathcal{A} \cup \{ Z_i \}) - f(\mathcal{A}) \geq f(\mathcal{B} \cup \{ Z_i \}) - f(\mathcal{B})$$ i.e. the marginal gain in mutual information by observing $Z_i$ is larger when the set of existing observations is smaller.

Main proof

Let $h = k/N$ where $N \in \mathbb{N}$. Define $Z_i \sim Poisson(h \theta)$ (conditionally independent given $\theta$). Let $Y_k = \sum_{i=1}^{N} Z_i$, $Y_{k+h} = \sum_{i=1}^{N+1} Z_i$, $Y_{k+2h} = \sum_{i=1}^{N+2} Z_i$. Then $Y_k \sim Poisson(k \theta)$, $Y_{k+1} \sim Poisson(k \theta + h)$, $Y_{k+2} \sim Poisson(k \theta + 2h)$.

The submodularity result above gives

$$g(k \theta + h) - g(k \theta) \geq g(k \theta + 2h) - g(k \theta + h),$$

which rearranges to

$$\frac{g(k \theta + 2h) - 2g(k \theta + h) + g(k \theta)}{h^2} \leq 0.$$

Taking the limit $N \to 0$ gives that $g''(k) \leq 0$ and thus $g$ is concave.


The proof just relies on an infinite divisibility property of the Poisson distribution, so the result will be true for other observation distributions also satisfying this.


Thanks to Iain Murray for helpful suggestions on relevant approaches for this proof.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.