3
$\begingroup$

EDIT

For two signals or random variables to be independent, the mutual information (MI) must be zero.

  1. What does mutual information (MI) convey?
  2. What is the meaning of independence? Does it mean different , unrelated?
$\endgroup$
  • $\begingroup$ I have deleted the first comment,but my question is not answered yet!My question is what is the implication of MI of error becoming minimum in terms of learning or parameter estimation where error defines the difference between the actual output and the model output. $\endgroup$ – Ria George Jul 27 '14 at 17:00
  • 1
    $\begingroup$ But your question was not answerable as it stood, since it relied on a false premise. The only correct response was to address the mistake and hope the question was edited to ask something else. Indeed, given your response to Aaron's explanation of your misunderstanding, it would have been dangerous to respond to your question (assuming anyone was able to guess your intent) until the problem was clear. $\endgroup$ – Glen_b -Reinstate Monica Jul 27 '14 at 17:08
  • $\begingroup$ I have edited the question & highlighted the problem clearly. What should I do now? Thank you for your guidelines $\endgroup$ – Ria George Jul 27 '14 at 17:13
  • $\begingroup$ Waiting for someone to answer would be the best policy, I think. I'd suggest while you wait for someone to answer, you review this and this and maybe this. $\endgroup$ – Glen_b -Reinstate Monica Jul 27 '14 at 17:17
  • $\begingroup$ @Glen_b: Fair enough! Seems completely warranted in that case. I guess it if happens again it might be good to provide a little context for others (if possible). I'll see what I can do about reviewing/editing :) $\endgroup$ – naught101 Feb 3 '16 at 12:44
4
$\begingroup$

If the mutual information is zero then the variables are independent. The closer the mutual information gets to zero the closer the variables are to being independent. If the MI is small then knowing X tells you little about X - Y.

Correlation is a linear operator. It is a weaker condition than independence. The error can be uncorrelated but nowhere close to independent.

$\endgroup$
  • $\begingroup$ If the MI is large than knowing X could tell you something about the error in your predictions. This information could theoretically be used to improve your predictions, i.e. your neural network is not exploiting all of the information available to it. $\endgroup$ – Aaron Jul 27 '14 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.