Lets say I have two independent random variables $T$ and $D$ both over finite integers sets $S_T$ and $S_D$ respectively.

Also, assume the probability function of $D$ is given to us and is only defined on non-negative integers, but the probability function of $T$ should be defined by us.

So we can assume the range of $T$ and $D$ are $\{-m_T,...,m_T\}$ and $\{0,...,m_D\}$ respectively.

Now I define another random variable "observed", as $O = T-D$ and would like to minimize the mutual information of $I(D; O)$ by defining the probability function of $T$.

I am trying to see whether it is possibly to make the mutual information zero? If not, what distribution $T$ should have to achieve the minimum mutual information.

  • $\begingroup$ The context of this questions is networking. Assume $D$ is the distance from a point on the Internet and your observation point and $T$ is a number called TTL that we choose at random in this case. Each node on the Internet decreases the TTL by one so if you observe a packet with TTL set to $t-d$, the goal is to guess the distance $d$, where $t$ and $d$ are drawn at random from $T$ and $D$ respectively. $\endgroup$ – Hooman Apr 3 '17 at 16:46
  • $\begingroup$ The terminology in this question is confusing. It sounds like you are using the term "probability space" to refer to the sets of values that can be attained by random variables--that is, their ranges. And how could either of these discrete variables possibly have probability "density" functions? $\endgroup$ – whuber Apr 3 '17 at 17:09
  • $\begingroup$ I fixed the terminology $\endgroup$ – Hooman Apr 3 '17 at 17:58

Mutual information of two random variables will be zero when the joint entropy is equal to the sum of the individual entropies... $$ I(D;O) = h(D)+h(O) - h(D,O), $$ so $$\mbox{if}\quad h(D,O) = h(D)+h(O) \quad \mbox{then} \quad I(D;O) = 0.$$

Now, the joint entropy will be the sum of the individual entropy values when the random variables are independent.

The trivial answer to your question (without more information about your probability mass functions (note: not density functions)) is that if you set $T = D$, then the mutual information $I(D;O)$ is trivially zero because $O$ is fixed and has zero entropy. I don't know how useful you'll find this answer, but yes, technically it is possible to make $I(D;O)$ zero by fixing $T$ to the value of $D$.

Beyond that, I think the question needs some more context.


Based on the context you added in your comment below your question, it sounds like $o\in O$ is an observable $\pm t\in T$ and $d\in D$ is a hidden variable, and you're trying to ask can I define $T$ to improve my estimate of $D$?

Think of my answer like this: if you don't know anything about $D$, you aren't going to get very far. If you know everything about $D$, then you can fix $T=D$ and the MI between $O$ and $D$ is $0$. Is there some middle way to specify $T$ to minimize $I(D;O)=I(D;T-D)$? I see you say that we hypothetically know the distribution of $D$. You've only told us the support of $D$ and I think we would also need to know something about the relationship between $O$ and $D$ to answer this. Without more information I can only suggest you start by simulating $D$ and then testing for distributions of $T$ that minimize the mutual info.

  • $\begingroup$ How's $O$ fixed and has zero entropy? We know that $O=T-D$ and even if $T$ and $D$ are drawn from the same distribution, the entropy​ of $O$ might not be zero. In fact, it's at least the entropy of $H(T)$. $\endgroup$ – Hooman Apr 3 '17 at 18:49
  • $\begingroup$ If $T=D$, then $O = 0$ no matter what. I'm not saying that $T$ and $D$ are drawn from the same distribution, I'm saying that you could specify that $T$ is literally fixed to always have the same value as $D$. Information entropy is a function of the probability of all possible states in the system: $h(X) = \sum_i p(X_i)\log p(X_i)$. If there is one possible state, then the entropy is zero because probability of that state is necessarily 1 and $\log(1) = 0$. $\endgroup$ – K Ball Apr 4 '17 at 14:19
  • $\begingroup$ @Hooman I don't have the rep to comment on your question, so I'm going to edit my response to include some of the meta context you put in the comments below your question. $\endgroup$ – K Ball Apr 4 '17 at 14:45
  • $\begingroup$ Your edit clarifies both the answer and the question a lot. $T$ and $D$ are independently drawn from two distributions and we only have the distribution of $D$ and would like to generate the distribution of $T$ such that MI of $O=T-D$ and $D$ is minimized. $\endgroup$ – Hooman Apr 5 '17 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.