# Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

268 questions
Filter by
Sorted by
Tagged with
6 views

### When is it difficult to estimate entropy and mutual information?

In what situations, or for what sort of data, is it challenging to estimate entropy, differential entropy and mutual information? or even infeasible? Is it hard only with small datasets and high ...
25 views

### What does maximizing mutual information do?

In information theory, there is something called the maximum entropy principle. Are other information measures, such as mutual information, also commonly maximized? If mutual information describes the ...
11 views

### How does the maximum entropy principle affect joint entropy, mutual information, and other info measures?

The maximum entropy principle says that we should use the probability distribution of a univariate dataset that has the highest level of entropy because it offers the lowest information. How does the ...
27 views

### Information Theoretical optimality of Jeffreys Prior?

Given a "conditional distribution" $f(x|\theta)$ and its corresponding Jeffreys prior $r(\theta)$, is there some information theoretic sense (in terms of quantities like mutual information) ...
51 views

### Interpretation of mutual information

Is it correct to say that the mutual information $I(X;Y)$ quantifies how well we can discriminate among the outcomes of $Y$ by looking at the outcomes of $X$ (or viceversa)? If yes, do you have any ...
10 views

### Social Network Analysis: assortativity between two classes and network structure

I have a network where nodes are labelled in two classes "a" and "b". I want to measure how connected these two groups are and I looked at assortativity by group. I want to use this measure in more ...
41 views

### Mutual information between a vector and a constant

Let's say that we have a tensor x sampled from its probability distribution, and that we have a constant vector c sampled from a degenerate distribution (its value is constant). Is the mutual ...
27 views

### Mutual information estimated on subsets of data

I am estimating the mutual information for a continuous data set using the kNN-based mutual information estimator proposed by Kraskov et al . Lets consider two features $X$ and $Y$, and the ...
30 views

### Interpretable General Measure of Dependence

I am looking for an interpretable measure between two random variables $X$ and $Y$ which quantifies the dependence between the two but does not assume linearity. Essentially, I am looking for a ...
15 views

### Advice on anomaly detection for a text

I have an idea of retreiving texts that have anomalous distribution of token frequency. Let's say I have a corpus of texts, and I build a document-term matrix based on token frequency. Naturally, ...
17 views

40 views

### Rewriting the mutual information of a linear model through conditional expectation

I am reading the following paper: http://web.mit.edu/18.325/www/telatar_capacity.pdf In this paper we have the following linear model, with $\mathbf{n}$ being additive noise: \begin{equation} \...
24 views

### Mutual Information as a Uncertainty Reduction Criteria For Arbitrary Field

I have been reviewing some papers such as this that uses Mutual Information (MI) as a criteria to obtain most informative point to approximate some large field an reduce uncertainty of the field. This ...
60 views

### Difference between Covariance and Mutual Information

I was going through the posts that describe the difference between covariance and MI and came across following from Quora The covariance of two random variables measures the strength of the linear ...
39 views

65 views

### How to choose sample size from probability density for computing mutual information based on continuous variables

I need to compute mutual information gain based two continuous variables $X$ and $Y$ $I(X|Y) = \int_X\int_Y p_{x.y}(x,y) \log(\frac{p_{x.y}(x,y)}{p_{x}(x)p_{y}(y)})$. I have used Kernel Density ...
55 views

### If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
131 views

### Non-negativity of interaction information for special trivariate case

Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies $$p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ),$$ for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
42 views

### Mutual Information between multi-dimensional and single dimensional Variables

I would like to estimate MI between two variables X and Y of shape (nXd) and ...
197 views

### Information Bottleneck principle of Deep Learning model implementation

I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other ...
39 views

### If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
So in a problem I'm working with, I'm essentially given 200 (arbitrary number) of samples from a joint distribution $P(x, y, z)$. However, what I want to be able to do is to extract the mutual ...