# Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

351 questions
Filter by
Sorted by
Tagged with
6 views

### Estimating conditional mutual informations from 2D histograms

I have binned marginal and joint distributions of two event features X and Y, i.e. p(X), p(Y) and p(X,Y) where the marginal distributions in X and Y are obtained by summing p(X,Y) over the bins of the ...
• 153
29 views

### Negative transfer entropy

By definition, transfer entropy cannot be negative. However, using the Kraskov estimator, negative values can be obtained. In general, should we take precautions to avoid getting negative values? How ...
• 11
1 vote
17 views

• 353
68 views

### Entropy Conditioned on A Certain Value

Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $C$. If $C$ is red, I shout the number zero. If $C$ is blue, I roll a fair die ...
• 161
28 views

### Conditional mutual information

I have three RVs X, Y, and T. Is the following equation true? I(X ; Y|T) = I(Y ; X|T) Can we express the conditional mutual information as: (X;Y|T) = I(X;Y) - I(X;Y;T) ?
• 31
1 vote
22 views

### Formula for conditional mutual information

What will be the formula for I(X;Y|Z,W)? Given: ...
• 31
54 views

• 141
143 views

### Why does collider adjustment in a shielded triplet tend to cause independence?

I created a causal model in which $X$ causes $Y$ and $Z$, and $Y$ causes $Z$ in the following way: ...
47 views

### Number of samples needed for same accuracy mutual information estimate

I'm trying to compare my work to another work that uses an discrete estimation of Mutual Information. I'll try to keep the example as short as possible. Let there be a population (n=1000) of solutions,...
171 views

### Estimating mutual information with gaussian kde between two continuous variables in python

I am trying to write a python code to estimate the mutual information between two continuous variables in python, using a gaussian kde to estimate the probability distributions. Checking it with ...
8 views

### Derivation of mutual information in Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design

I am troubled by how the following expression is derived from section 2.2 of Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design (https://arxiv.org/pdf/0912.3995.pdf)...
• 223
10 views

### Relating difference in distribution of dataset to information gain

I wonder if there are ways to relate difference in distribution of dataset to information gain. For example, I train a model on dataset D_1 and obtain a trained model M_1 and train a model on dataset ...
• 223
39 views

### Do we need to normalize all the variables before calculating mutual information for variables using sklearn?

In calculating the mutual information on sklearn using either mutual_info_classif or mutual_info_regression, the underlying algorithm seems to use KNN to derive the mutual information for the variable....
• 135
1 vote
161 views

### mutual information and edge weights in a bayesian network

The mutual information between two random variables X and Y can be stated formally as follows: I(X ; Y) = H(X) – H(X | Y) Where I(X ; Y) is the mutual information for X and Y, H(X) is the entropy for ...
• 73
1 vote
28 views

### Which method is best to find the correlation between 2 datasets in my case

I have multiple datasets that i need to find the correlation between them The problem is that my datasets are mainly zeros and ones (zero means patient does not have the disease and 1 means patient ...
• 249
66 views

### Maximizing the mutual information related to minimizing KL divergence?

Let's say we are drawing two random variables from two distributions, like x~p and z~q, then maximizing mutual information I(x;z) leads to decreasing KL divergence D_kl(p|q) ? This sounds correct to ...
• 11
1 vote
23 views

### Mutual Information in the presence of noise

Let Y=X+N, where N is the noise distribution. a. A user can observe only Y. (I(X; Y)=?) b. A user can estimate the mean of N and he can observe Y only. (I(X;Y|mean(N))) He has no information regarding ...
• 31
26 views

### Bayesian estimation of mutual information for continuous variables

I am looking for references that derive a Bayesian estimator for mutual information in the case of continuous data. My understanding is that this has been explored for discrete (e.g. Hutter, 2002, ...
1 vote
41 views

### Zero mutual information

I have two random variables X and Y=X+N. What should be the condition such that I(X; Y)=0? Can anyone direct me towards the relevant references?
• 31
34 views

### Infinite mutual information for continuous multidimensional random variables

I am struggling with understanding the notion of the infinity of the mutual information. Say, one has an multidimensional continuous Gaussian variable $(X_1, X_2...X_d)$. Why would the mutual ...
• 111
22 views

### Determining history length in Transfer Entropy

Supposing I have a time series observation corresponding to 5 sensors, and I wish to find the causation(Transfer entropy) between all pairs of nodes, how do I go about choosing the history length ...
• 195
24 views

### Expansion of mutual information in terms of moments

Consider two random variables $X,Y$ (possibly multi-dimensional). The mutual information is defined by: $$I(X,Y) = \sum_{x,y} P(x,y)\ln\left(\frac{P(x,y)}{P(x)P(y)}\right)$$ where $P(x,y)$ is the ...
• 3,348
7 views

### How to calculate the uncertainty of a variable explained by other variables

I'm working on implementing a method from a paper for solving the FRn-k problem for a specific type of biological data. [FRn−k]: Given an initial set of n features, find the subset with k<n ...
• 111
53 views

### how the lemma is applied on InfoGAN paper

I am reading the InfoGAN paper and I can understand how the lemma (attached below) is proved. The original paper uses the lemma in this way: first it finds out that (equation 4 in the paper) I can ...
25 views

### Finding an appropriate formula to measure mutual information between pairs of observations, with K independent features

I am currently looking for some variant of mutual information, which I can use for the following setup: I have an N (subjects) x K (features) matrix: each column (feature) follows roughly N(0,1). (...
• 23
1 vote
237 views

### What to Choose and Why ? Chi-Square or Mutual Information for Categorical Feature Selection

I am working on an ML problem. The dataset is of shape (15036, 216) containing all categorical variables. The task is to select the top 10 features which are applicable using any two feature selection ...
• 11
9 views

### Does the number of conditional variables affect the resulf of mutual information

I have a question about whether the mutual information be consistent with the number of conditional variables or not? For example, would the value of mutual information between the two variables $X$ ...
• 1,032
81 views

### Mutual information of linearly transformed normal variable

Given that we have a variable $\boldsymbol{x}{\sim}\mathcal{N}(\boldsymbol{0},\boldsymbol{I}); \boldsymbol{x}\in\mathbb{R}^{F}$ where $F$ is the input size and we apply an affine transformation ...
30 views

### Mutual Information between Intersecting Sets

Consider that we have a user that can make queries to a database. When a user specifies a query, they can't see what exact data-points the query filters, but they can only see the result of the query. ...
15 views

• 131
30 views

### Why mutual information gives me negative results

I have the following table: ...
• 187
32 views

### How to determine the strength of the mutual information value

I am reading about mutual information dependency measures. I found that it ranges from $0$ to infinity. I have one simple question. How can I measure the strength of the dependency using this method? ...
• 1,032
122 views

### If Mutual Information is $\text{KLD}(P(X, Y) || P(X)P(Y))$, why is $\text{KLD}(P(X)P(Y) || P(X, Y))$ never mentioned/used?

Mutual Information is defined as the Kullback-Liebler Divergence between $P(X, Y)$ and $P(X)P(Y)$. Using definitions of KLD, this can be intuitively understood as the cost in average code length if we ...
• 43
15 views

### Is there a measure of association that is not a concave function of mutual information?

I'm reading Mutual Dependence: A Novel Method for Computing Dependencies Between Random Variables and I see in figure 2 and figure 3 that correlation, correlation distance, and mutual dependence are ...
• 3,383
113 views

### mutual information and maximual infomation coffecient

I am interested in calculating the strength between random variables. I found that the maximal information coefficient is one of the good methods to use and it is robust to the mutual information ...
• 1,032
35 views

### Conditional mutual information conditioned on event

Consider discrete random variables $Z$, $W$, $X$, and some event $\mathcal{E}$: I'm quite confused about the meaning of the conditional mutual information $I[ Z : W \mid X, \mathcal{E} ]$. I'm aware ...
1 vote
81 views

### Does Sample Size affects Mutual Information for Feature Selection?

There is a dataset with n rows (samples) and p columns (variables/features), the objective is to predict a certain target variable (y). Should n (sample size) matter to the results of pairwise mutual ...
• 13
29 views

### Can Mutual Information based feature selection be used when the input variables are numerical and the output is categorical?

I am working on a machine learning project for a classification problem. In the dataset the input variables are numerical and the output is categorical. Is it appropriate to apply the Mutual ...
1 vote
93 views

### Inuition for Aggregators in Adjusted Mutual Information for Clustering

A post comparing Adjusted Rand Index (ARI) and Adjusted Mutual Information (AMI) cites the rule of thumb that we should use ARI when the ground truth clustering has large equal sized clusters use AMI ...
• 521
1 vote
46 views

### Mutual information equality based on chain rule

I have an introduction to information theory course which states the following equality : $I(X_1; Y_1; Y_2) = I(X_1; Y_1) + I(X_1; Y_2 | Y_1)$ ​ The professor claims this can be proved with the chain ...
EDIT: By noodling around a bit, I found that they can actually both be re-arranged into even more similar forms: $Z = -\frac{1}{2}(\ln(1-\rho) - \ln(1+\rho))$ \$I =-\frac{1}{2}(\ln(1-\rho) + \ln(1+\rho)...