I am trying to solve this problem (no one has yet answered, though the comment by Xi'an says it is simple!)

entropy: is H(X+Y) = H(X) + H(X+Y)|X) true?

In writing out the entropy expression, one of the probabilities is $P(X+Y|X)$. Does this expand or simplify to any other expression that could be used?

(I do not think so..., but tell me your answer)

It is not for a class. Reading a textbook (Cover Information theory), I am just stuck on this small pirnt.

  • $\begingroup$ Try writing it out more explicitly... $P(X+Y = y|X=x)$. Do you see the simplification now? Hint: You can write this in terms of $P(Y = ?). $\endgroup$ – knrumsey Apr 14 '18 at 20:35
  • $\begingroup$ I will work on this and come back. $\endgroup$ – isolatedstudent Apr 15 '18 at 6:56
  • $\begingroup$ So the answer by Xi'an to the other post about entropy (linked above) gives the answer to this question inside the answer to that question. However, though I understand the overall idea, there are two formal details that are new to me. Since they are the same for this question I would like to ask about these small details. I will show the proof, and my questions, but tomrrow. $\endgroup$ – isolatedstudent Apr 17 '18 at 7:11

As mentioned by knrumsey in the comment to the question you are conditioning on $X$ being a known quantity, effectively making $X$ a constant $X=x$.

Understanding the notation matters a lot here. The quantity $X$ is a random variable and $x$ is the value that $X$ equals in this scenario. The comment differs slightly in statement from the linked post, in the linked post the equation is written $\Pr(X+Y=x+y|X=x).$ Now we know that this is equivalently, $\Pr(X+Y=x+y|X=x)=\Pr(x+Y=y+x|X=x)$ and this second expression for the event ${x+Y=y+x|X=x}$ can subtract the $x$ from each side, because this is a constant in this event. Writing the resulting event is then $\Pr(x+Y=y+x|X=x)=\Pr(Y=y| X=x)$ which is the claim in the underbrace in Xi'an's post.

  • $\begingroup$ Looks like you did not finish editing (maybe clean up?), but I think I understand. Restatement: since X=x is a constant, you can change X to x on the left side of the conditional, then it is just a constant offset that "shifts" the value of Y, and does not affect the probability distribution of Y. Is that right? $\endgroup$ – isolatedstudent Jun 17 '18 at 22:45
  • $\begingroup$ @isolatedstudent yes, the rest shouldn't of updated in the post, I'll remove the dangling language. If this clarifies please consider marking as correct. $\endgroup$ – Lucas Roberts Jun 17 '18 at 23:31
  • $\begingroup$ i have learned something valuable. $\endgroup$ – isolatedstudent Jun 26 '18 at 6:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.