EDIT
For two signals or random variables to be independent, the mutual information (MI) must be zero.
- What does mutual information (MI) convey?
- What is the meaning of independence? Does it mean different , unrelated?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ I have deleted the first comment,but my question is not answered yet!My question is what is the implication of MI of error becoming minimum in terms of learning or parameter estimation where error defines the difference between the actual output and the model output. $\endgroup$– Ria GeorgeCommented Jul 27, 2014 at 17:00
-
1$\begingroup$ But your question was not answerable as it stood, since it relied on a false premise. The only correct response was to address the mistake and hope the question was edited to ask something else. Indeed, given your response to Aaron's explanation of your misunderstanding, it would have been dangerous to respond to your question (assuming anyone was able to guess your intent) until the problem was clear. $\endgroup$– Glen_bCommented Jul 27, 2014 at 17:08
-
$\begingroup$ I have edited the question & highlighted the problem clearly. What should I do now? Thank you for your guidelines $\endgroup$– Ria GeorgeCommented Jul 27, 2014 at 17:13
-
$\begingroup$ Waiting for someone to answer would be the best policy, I think. I'd suggest while you wait for someone to answer, you review this and this and maybe this. $\endgroup$– Glen_bCommented Jul 27, 2014 at 17:17
-
$\begingroup$ @Glen_b: Fair enough! Seems completely warranted in that case. I guess it if happens again it might be good to provide a little context for others (if possible). I'll see what I can do about reviewing/editing :) $\endgroup$– naught101Commented Feb 3, 2016 at 12:44
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
1
If the mutual information is zero then the variables are independent. The closer the mutual information gets to zero the closer the variables are to being independent. If the MI is small then knowing X tells you little about X - Y.
Correlation is a linear operator. It is a weaker condition than independence. The error can be uncorrelated but nowhere close to independent.
-
$\begingroup$ If the MI is large than knowing X could tell you something about the error in your predictions. This information could theoretically be used to improve your predictions, i.e. your neural network is not exploiting all of the information available to it. $\endgroup$– AaronCommented Jul 27, 2014 at 17:30