# Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

255 questions
Filter by
Sorted by
Tagged with
10 views

### Social Network Analysis: assortativity between two classes and network structure

I have a network where nodes are labelled in two classes "a" and "b". I want to measure how connected these two groups are and I looked at assortativity by group. I want to use this measure in more ...
39 views

### Mutual information between a vector and a constant

Let's say that we have a tensor x sampled from its probability distribution, and that we have a constant vector c sampled from a degenerate distribution (its value is constant). Is the mutual ...
24 views

### Mutual information estimated on subsets of data

I am estimating the mutual information for a continuous data set using the kNN-based mutual information estimator proposed by Kraskov et al . Lets consider two features $X$ and $Y$, and the ...
27 views

### Interpretable General Measure of Dependence

I am looking for an interpretable measure between two random variables $X$ and $Y$ which quantifies the dependence between the two but does not assume linearity. Essentially, I am looking for a ...
14 views

### Advice on anomaly detection for a text

I have an idea of retreiving texts that have anomalous distribution of token frequency. Let's say I have a corpus of texts, and I build a document-term matrix based on token frequency. Naturally, ...
16 views

40 views

### Rewriting the mutual information of a linear model through conditional expectation

I am reading the following paper: http://web.mit.edu/18.325/www/telatar_capacity.pdf In this paper we have the following linear model, with $\mathbf{n}$ being additive noise: \begin{equation} \...
24 views

### Mutual Information as a Uncertainty Reduction Criteria For Arbitrary Field

I have been reviewing some papers such as this that uses Mutual Information (MI) as a criteria to obtain most informative point to approximate some large field an reduce uncertainty of the field. This ...
54 views

### Difference between Covariance and Mutual Information

I was going through the posts that describe the difference between covariance and MI and came across following from Quora The covariance of two random variables measures the strength of the linear ...
37 views

29 views

### Finding related variables

I have read this and this question, but non of them answers my question properly. I am not very familiar with concepts of stat, so please bear with me. I have 2 types of variables, type A (circles) ...
64 views

### How to choose sample size from probability density for computing mutual information based on continuous variables

I need to compute mutual information gain based two continuous variables $X$ and $Y$ $I(X|Y) = \int_X\int_Y p_{x.y}(x,y) \log(\frac{p_{x.y}(x,y)}{p_{x}(x)p_{y}(y)})$. I have used Kernel Density ...
55 views

### If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
129 views

### Non-negativity of interaction information for special trivariate case

Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies $$p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ),$$ for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
41 views

### Mutual Information between multi-dimensional and single dimensional Variables

I would like to estimate MI between two variables X and Y of shape (nXd) and ...
183 views

### Information Bottleneck principle of Deep Learning model implementation

I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other ...
39 views

### If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
63 views

### Choosing appropriate statistical tests

I am trying to run stats on medical treatment data and would like your guys and girls help Goal: Comparing two trends of annual percentages for statistical significance. Is one decreasing due to the ...
149 views

### How to normalize mutual information between to real-valued random variables?

How can I normalize mutual information between to real-valued random variables using Python or R? ...