Consider $m$ channels indexed by $i$ with $1 \leq i \leq m$. The input alphabets are from the same finite set $\mathcal{X}$. Let $\pi$ denote a probability distribution on $\mathcal{X}$. Define the function \begin{equation} f_i(\pi)=I(X:Y_i) \end{equation} where $X \sim \pi$ and given $X=x$, $Y_i$ is generated as $Y_i|X=x \sim Y|i,x$, that is we send the input $X$ to the $i^{th}$ channel and observe the output $Y_i$. $I(.,.)$ is the mutual information between two random variables. Further, define $f(\pi)=max_i f_i(\pi)$.

Now suppose we saw an output $Y_j$ from a fixed channel $j$. Assuming a prior $\pi_0$ on $\mathcal{X}$ we denote the posterior after seeing $Y_j$ as $\pi_{1,j}$. Is it true that: \begin{equation} f(\pi_0) \geq E[f(\pi_{1,j})] \end{equation} for all $j \in [m]$? Note the expectation is taken over the randomness of $Y_j$ and $X$.

It is easily seen that : \begin{equation} f_i(\pi_0) \geq E[f_i(\pi_{1,j})] \end{equation} by concavity of mutual information and the fact the expectation of posterior is just the prior. Unfortunately the $max$ operation destroys this concavity for $f$. But the inequality seems intuitive in that the maximum information you can extract about the input from any channel decreases as the number of observations accumulate.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.