In information theory, there is something called the maximum entropy principle. Are other information measures, such as mutual information, also commonly maximized? If mutual information describes the reduction in uncertainty of characterizing one random variable (r.v. 1) based on full knowledge of a second random variable (r.v. 2), then would maximizing mutual information mean knowing everything about r.v. 2 will allow us to also have full knowledge of r.v. 1?


Just to give one example in machine learning context, for unsupervised learning, maximizing $I(X, Y)$ can build good representations $Y$ of input $X$, see here and here

  • 3
    $\begingroup$ Mutual information is a measure of the amount of information one variable reveals about another where 0 indicates independence, larger values indicate stronger dependence. It always helps to go back to the source, in this case, Gelfand and Yaglom, Calculation of amount of information about a random function contained in another such function, American Mathematical Society Translations, 1957, Series 2. 12: 199–246. Also, there's this thread..stats.stackexchange.com/questions/81659/… $\endgroup$ Jul 31 '20 at 12:42
  • $\begingroup$ yeah, I totally agree, and my answer isn't a proper one and needs refinement. I just wanted to quickly comment on what maximizing MI can do. @MikeHunter $\endgroup$
    – doubllle
    Jul 31 '20 at 12:57
  • $\begingroup$ if elements in the mutual information matrix are nothing more than measures of dependence between variables in a multivariate dataset, similar to the correlation matrix, than how does maximizing the mutual information matrix differ from maximizing the correlation matrix? Is it simply able to pick out non-linear dependencies that the correlation matrix can't? $\endgroup$
    – develarist
    Aug 4 '20 at 8:57
  • 1
    $\begingroup$ Being able to pick up nonlinear correlation is nice, isn't it? However, maximum of MI is hard to obtain due to the joint probability term, so the linked papers are maximizing its lower bound instead in practice. So practically, it is hard to fully recover $X$ from $Y$. $\endgroup$
    – doubllle
    Aug 4 '20 at 10:04
  • $\begingroup$ @doubllle can you describe what the maximization of mutual information accomplishes for the two attached papers in your answer? $\endgroup$
    – develarist
    Sep 7 '20 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.