# Posterior entropy behaviour for Poisson model

Suppose $$Y \sim Poisson(k \theta)$$ is observed (number of events), where $$k>0$$ is a constant (length of observation time) and $$\theta$$ is an unknown parameter (event rate) with some prior distribution.

Is it possible to show that the expectation (with respect to $$Y$$ and $$\theta$$) of the Shannon entropy of the posterior is convex in $$k$$? i.e. the expected information gain is subject to decreasing returns in the length of observation time.

I'm interested in results for any particular choice of prior (except a point mass) or for all priors.

Answer based on the original question.

The Shannon entropy of the Poisson distribution is

$$\lambda [1-\log(\lambda )]+e^{-\lambda }\sum _{k=0}^{\infty }{\frac {\lambda ^{k}\log(k!)}{k!}},$$ where $$\lambda = \theta k$$ in your case. For each value of $$\theta$$, this seems to be a concave function of $$k$$:

# Approximation to the Shannon Entropy truncating the series at N
shan.pois <- Vectorize(function(k){
x <- 1:N
val <- (theta*k)*(1-log(theta*k)) + sum(  dpois(x, lambda = theta*k)*lgamma(x+1) )
return(val)
})

# Plot for different values of theta
N <- 1000

theta <- 1
curve(shan.pois,0,10, n = 1000, ylim = c(0,5),lwd=2, xlab = "k", ylab = "S(k)")

thetas <- seq(0.1,10,by = 0.1)
for(i in 1:length(thetas)){
theta <- thetas[i]
curve(shan.pois,0,10, n = 1000, add = T, col = "grey")
}


So, even if you find a strongly informative prior that, for a specific sample, the posterior expectation is convex, it will eventually become concave by the concentration properties of the posterior distribution.

• Thanks for the answer! Unless I'm missing a connection, I think you're looking at expectation of the entropy of y wrt the posterior distribution, but I'm interested in expectation of the entropy of the posterior distribution with respect to y and $\theta$. I'll reword the question to make it clearer. – Dennis Prangle Apr 24 '20 at 14:08
• @DennisPrangle That is certainly a very different question! I will have a go. – Typo Apr 24 '20 at 14:10

### Information theory representation

The expectation with respect to $$Y$$ of the posterior entropy is the conditional entropy $$H(\theta | Y) = H(\theta) - \mathrm{I}(Y; \theta)$$ where $$H(\theta)$$ is prior entropy and $$I$$ is mutual information.

Let $$g(k)$$ denote the mutual information as a function of $$k$$. So the question can be rephrased as showing that $$g$$ is concave.

## Submodularity of mutual information (discrete case)

The proof extends a similar result on mutual information under repeated observations.

Let $$\mathcal{S} = \{ Z_1, Z_2,\ldots, Z_n \}$$, a set of conditionally independent (given $$\theta$$) random variables with the same distribution. Define $$f(\mathcal{A}) = \mathrm{I}(\mathcal{A}; \theta)$$ where $$\mathcal{A} \subseteq \mathcal{S}$$.

Proposition 2 of Krause and Guestrin shows that $$f(A)$$ is a submodular function (the proof follows from submodularity of entropy).

(Edit: in fact a modification of this proposition is needed to deal with the fact that $$\theta$$ is a continuous parameter. See the proof of equation (25) in Appendix G of Prangle, Harbisher and Gillespie.)

That is, for every $$\mathcal{A} \subseteq \mathcal{B} \subseteq \mathcal{S}$$ and $$Z_i \in \mathcal{S} \setminus \mathcal{B}$$: $$f(\mathcal{A} \cup \{ Z_i \}) - f(\mathcal{A}) \geq f(\mathcal{B} \cup \{ Z_i \}) - f(\mathcal{B})$$ i.e. the marginal gain in mutual information by observing $$Z_i$$ is larger when the set of existing observations is smaller.

## Main proof

Let $$h = k/N$$ where $$N \in \mathbb{N}$$. Define $$Z_i \sim Poisson(h \theta)$$ (conditionally independent given $$\theta$$). Let $$Y_k = \sum_{i=1}^{N} Z_i$$, $$Y_{k+h} = \sum_{i=1}^{N+1} Z_i$$, $$Y_{k+2h} = \sum_{i=1}^{N+2} Z_i$$. Then $$Y_k \sim Poisson(k \theta)$$, $$Y_{k+1} \sim Poisson(k \theta + h)$$, $$Y_{k+2} \sim Poisson(k \theta + 2h)$$.

The submodularity result above gives

$$g(k \theta + h) - g(k \theta) \geq g(k \theta + 2h) - g(k \theta + h),$$

which rearranges to

$$\frac{g(k \theta + 2h) - 2g(k \theta + h) + g(k \theta)}{h^2} \leq 0.$$

Taking the limit $$N \to 0$$ gives that $$g''(k) \leq 0$$ and thus $$g$$ is concave.

### Extension

The proof just relies on an infinite divisibility property of the Poisson distribution, so the result will be true for other observation distributions also satisfying this.

### Credit

Thanks to Iain Murray for helpful suggestions on relevant approaches for this proof.