Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Estimating conditional mutual informations from 2D histograms

I have binned marginal and joint distributions of two event features X and Y, i.e. p(X), p(Y) and p(X,Y) where the marginal distributions in X and Y are obtained by summing p(X,Y) over the bins of the ...
user avatar
0 votes
0 answers
29 views

Negative transfer entropy

By definition, transfer entropy cannot be negative. However, using the Kraskov estimator, negative values can be obtained. In general, should we take precautions to avoid getting negative values? How ...
user avatar
  • 11
1 vote
0 answers
17 views

Understanding continuous variable entropy

I am struggling to understand continuous variable entropies and mutual informations for 2 or more variables. Consider 2D normal distribution $\rho(x,y)$ defined as follows $$X\sim\mathcal{N}(0, \...
user avatar
2 votes
2 answers
70 views

MSE is 'scale dependent'. R-squared seems a better measure of fit for regressions. Are there others?

Mean-Squared Error is scale dependent. For example if I have an MSE of 0.1 and multiply all of X and Y by 100, redo my regression and calculate MSE, I get an MSE of 1000.0. ((y_true-y_regr)^2 ---> ...
user avatar
  • 21
2 votes
2 answers
102 views

Why is mutual information symmetric?

I know that mutual information is the Kullback-Leibler divergence between $p(x,y)$ and $p(x)p(y)$. But mutual information is also described as the amount of entropy lost (or, in another sense, the ...
user avatar
  • 147
1 vote
1 answer
87 views

Why does mutual information use KL divergence?

Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as $$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\...
user avatar
4 votes
1 answer
68 views

Entropy Conditioned on A Certain Value

Consider this example: I have a box with 10 balls in it, 9 red and 1 blue. I take a ball randomly. Let's call the color $C$. If $C$ is red, I shout the number zero. If $C$ is blue, I roll a fair die ...
user avatar
  • 161
0 votes
1 answer
28 views

Conditional mutual information

I have three RVs X, Y, and T. Is the following equation true? I(X ; Y|T) = I(Y ; X|T) Can we express the conditional mutual information as: (X;Y|T) = I(X;Y) - I(X;Y;T) ?
user avatar
1 vote
1 answer
22 views

Formula for conditional mutual information

What will be the formula for I(X;Y|Z,W)? Given: ...
user avatar
-1 votes
1 answer
54 views

How does mutual information equal $I(x;y)=\sum_{x,y}p(x,y)log(\frac{p(x|y)}{p(x)})$

How does mutual information equal $I(x;y)=\sum_{x,y}p(x,y)log(\frac{p(x|y)}{p(x)})$? So the definition for mutual information I'm familiar with is $I(X;Y)=H(X)-H(X|Y)$. Where $H(X)=-\sum_{x\in X}p(x)...
user avatar
0 votes
0 answers
30 views

Example of calculating the mutual information of an input x with the task label y AKA I(x;y)?

I'm reading a paper currently about contrastive learning and they were mentioning that ideally you want your encoders to learn the minimal amount of information required for a task i.e. I(x;y)=I(E(x);...
user avatar
2 votes
0 answers
22 views

Intuition behind redundant information

Suppose we have a set of variables, we call them "sources" ${\bf S} = \{S_1,\ldots,S_n\}$, and a "target" variable, denoted with $Y$. A measure of overall dependency between ${\bf ...
user avatar
7 votes
2 answers
143 views

Why does collider adjustment in a shielded triplet tend to cause independence?

I created a causal model in which $X$ causes $Y$ and $Z$, and $Y$ causes $Z$ in the following way: ...
user avatar
0 votes
1 answer
47 views

Number of samples needed for same accuracy mutual information estimate

I'm trying to compare my work to another work that uses an discrete estimation of Mutual Information. I'll try to keep the example as short as possible. Let there be a population (n=1000) of solutions,...
user avatar
0 votes
0 answers
171 views

Estimating mutual information with gaussian kde between two continuous variables in python

I am trying to write a python code to estimate the mutual information between two continuous variables in python, using a gaussian kde to estimate the probability distributions. Checking it with ...
user avatar
0 votes
0 answers
8 views

Derivation of mutual information in Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design

I am troubled by how the following expression is derived from section 2.2 of Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design (https://arxiv.org/pdf/0912.3995.pdf)...
user avatar
  • 223
0 votes
0 answers
10 views

Relating difference in distribution of dataset to information gain

I wonder if there are ways to relate difference in distribution of dataset to information gain. For example, I train a model on dataset D_1 and obtain a trained model M_1 and train a model on dataset ...
user avatar
  • 223
0 votes
0 answers
39 views

Do we need to normalize all the variables before calculating mutual information for variables using sklearn?

In calculating the mutual information on sklearn using either mutual_info_classif or mutual_info_regression, the underlying algorithm seems to use KNN to derive the mutual information for the variable....
user avatar
  • 135
1 vote
1 answer
161 views

mutual information and edge weights in a bayesian network

The mutual information between two random variables X and Y can be stated formally as follows: I(X ; Y) = H(X) – H(X | Y) Where I(X ; Y) is the mutual information for X and Y, H(X) is the entropy for ...
user avatar
1 vote
1 answer
28 views

Which method is best to find the correlation between 2 datasets in my case

I have multiple datasets that i need to find the correlation between them The problem is that my datasets are mainly zeros and ones (zero means patient does not have the disease and 1 means patient ...
user avatar
  • 249
0 votes
0 answers
66 views

Maximizing the mutual information related to minimizing KL divergence?

Let's say we are drawing two random variables from two distributions, like x~p and z~q, then maximizing mutual information I(x;z) leads to decreasing KL divergence D_kl(p|q) ? This sounds correct to ...
user avatar
  • 11
1 vote
0 answers
23 views

Mutual Information in the presence of noise

Let Y=X+N, where N is the noise distribution. a. A user can observe only Y. (I(X; Y)=?) b. A user can estimate the mean of N and he can observe Y only. (I(X;Y|mean(N))) He has no information regarding ...
user avatar
0 votes
0 answers
26 views

Bayesian estimation of mutual information for continuous variables

I am looking for references that derive a Bayesian estimator for mutual information in the case of continuous data. My understanding is that this has been explored for discrete (e.g. Hutter, 2002, ...
user avatar
1 vote
1 answer
41 views

Zero mutual information

I have two random variables X and Y=X+N. What should be the condition such that I(X; Y)=0? Can anyone direct me towards the relevant references?
user avatar
0 votes
0 answers
34 views

Infinite mutual information for continuous multidimensional random variables

I am struggling with understanding the notion of the infinity of the mutual information. Say, one has an multidimensional continuous Gaussian variable $(X_1, X_2...X_d)$. Why would the mutual ...
user avatar
  • 111
0 votes
0 answers
22 views

Determining history length in Transfer Entropy

Supposing I have a time series observation corresponding to 5 sensors, and I wish to find the causation(Transfer entropy) between all pairs of nodes, how do I go about choosing the history length ...
user avatar
0 votes
0 answers
24 views

Expansion of mutual information in terms of moments

Consider two random variables $X,Y$ (possibly multi-dimensional). The mutual information is defined by: $$ I(X,Y) = \sum_{x,y} P(x,y)\ln\left(\frac{P(x,y)}{P(x)P(y)}\right) $$ where $P(x,y)$ is the ...
user avatar
  • 3,348
0 votes
0 answers
7 views

How to calculate the uncertainty of a variable explained by other variables

I'm working on implementing a method from a paper for solving the FRn-k problem for a specific type of biological data. [FRn−k]: Given an initial set of n features, find the subset with k<n ...
user avatar
  • 111
0 votes
0 answers
53 views

how the lemma is applied on InfoGAN paper

I am reading the InfoGAN paper and I can understand how the lemma (attached below) is proved. The original paper uses the lemma in this way: first it finds out that (equation 4 in the paper) I can ...
user avatar
0 votes
0 answers
25 views

Finding an appropriate formula to measure mutual information between pairs of observations, with K independent features

I am currently looking for some variant of mutual information, which I can use for the following setup: I have an N (subjects) x K (features) matrix: each column (feature) follows roughly N(0,1). (...
user avatar
  • 23
1 vote
0 answers
237 views

What to Choose and Why ? Chi-Square or Mutual Information for Categorical Feature Selection

I am working on an ML problem. The dataset is of shape (15036, 216) containing all categorical variables. The task is to select the top 10 features which are applicable using any two feature selection ...
user avatar
  • 11
0 votes
0 answers
9 views

Does the number of conditional variables affect the resulf of mutual information

I have a question about whether the mutual information be consistent with the number of conditional variables or not? For example, would the value of mutual information between the two variables $X$ ...
user avatar
  • 1,032
0 votes
1 answer
81 views

Mutual information of linearly transformed normal variable

Given that we have a variable $\boldsymbol{x}{\sim}\mathcal{N}(\boldsymbol{0},\boldsymbol{I}); \boldsymbol{x}\in\mathbb{R}^{F}$ where $F$ is the input size and we apply an affine transformation ...
user avatar
0 votes
0 answers
30 views

Mutual Information between Intersecting Sets

Consider that we have a user that can make queries to a database. When a user specifies a query, they can't see what exact data-points the query filters, but they can only see the result of the query. ...
user avatar
  • 1
0 votes
0 answers
15 views

InfoVAE - Full loss implementation

studying the paper "InfoVAE: Information Maximizing Variational Autoencoders"[1], I got confused about the full loss function presented as following: $\mathcal{L}_{\text{InfoVAE}} = \mathbb{...
user avatar
  • 101
3 votes
1 answer
83 views

Mutual information is not zero for independent variables and negative for weakly dependency

To the best of my knowledge, mutual information (MI) is zero if and only if the variables are independent. I have simulated copula data and computed the ...
user avatar
  • 1,032
0 votes
0 answers
84 views

normalized conditional mutual information

I am learning mutual information and need to apply it to my data. I strongly need to find a normalized conditional mutual information to have a range within [0,1] instead of 0 to infinity in ...
user avatar
  • 1,032
2 votes
1 answer
139 views

How to do basic causal inference for two random variables [closed]

Suppose that we have two random variables $X$ and $Y$. Is it possible to detect the dependency $Y = f(X)$ for some arbitrary $f$? Is there a way of quantifying if $Y$ is "nearly equal" to $f(...
user avatar
  • 131
0 votes
0 answers
30 views

Why mutual information gives me negative results

I have the following table: ...
user avatar
  • 187
2 votes
0 answers
32 views

How to determine the strength of the mutual information value

I am reading about mutual information dependency measures. I found that it ranges from $0$ to infinity. I have one simple question. How can I measure the strength of the dependency using this method? ...
user avatar
  • 1,032
4 votes
1 answer
122 views

If Mutual Information is $\text{KLD}(P(X, Y) || P(X)P(Y))$, why is $\text{KLD}(P(X)P(Y) || P(X, Y))$ never mentioned/used?

Mutual Information is defined as the Kullback-Liebler Divergence between $P(X, Y)$ and $P(X)P(Y)$. Using definitions of KLD, this can be intuitively understood as the cost in average code length if we ...
user avatar
  • 43
0 votes
0 answers
15 views

Is there a measure of association that is not a concave function of mutual information?

I'm reading Mutual Dependence: A Novel Method for Computing Dependencies Between Random Variables and I see in figure 2 and figure 3 that correlation, correlation distance, and mutual dependence are ...
user avatar
  • 3,383
4 votes
1 answer
113 views

mutual information and maximual infomation coffecient

I am interested in calculating the strength between random variables. I found that the maximal information coefficient is one of the good methods to use and it is robust to the mutual information ...
user avatar
  • 1,032
0 votes
0 answers
35 views

Conditional mutual information conditioned on event

Consider discrete random variables $Z$, $W$, $X$, and some event $\mathcal{E}$: I'm quite confused about the meaning of the conditional mutual information $I[ Z : W \mid X, \mathcal{E} ]$. I'm aware ...
user avatar
1 vote
0 answers
81 views

Does Sample Size affects Mutual Information for Feature Selection?

There is a dataset with n rows (samples) and p columns (variables/features), the objective is to predict a certain target variable (y). Should n (sample size) matter to the results of pairwise mutual ...
user avatar
0 votes
0 answers
29 views

Can Mutual Information based feature selection be used when the input variables are numerical and the output is categorical?

I am working on a machine learning project for a classification problem. In the dataset the input variables are numerical and the output is categorical. Is it appropriate to apply the Mutual ...
user avatar
  • 1
1 vote
0 answers
93 views

Inuition for Aggregators in Adjusted Mutual Information for Clustering

A post comparing Adjusted Rand Index (ARI) and Adjusted Mutual Information (AMI) cites the rule of thumb that we should use ARI when the ground truth clustering has large equal sized clusters use AMI ...
user avatar
  • 521
1 vote
1 answer
46 views

Mutual information equality based on chain rule

I have an introduction to information theory course which states the following equality : $I(X_1; Y_1; Y_2) = I(X_1; Y_1) + I(X_1; Y_2 | Y_1)$ ​ The professor claims this can be proved with the chain ...
user avatar
2 votes
1 answer
105 views

Z-transforming Pearson correlation vs. converting to mutual information: they seem to be related, but how?

EDIT: By noodling around a bit, I found that they can actually both be re-arranged into even more similar forms: $Z = -\frac{1}{2}(\ln(1-\rho) - \ln(1+\rho))$ $I =-\frac{1}{2}(\ln(1-\rho) + \ln(1+\rho)...
user avatar
3 votes
0 answers
135 views

Conditional Mutual information in R ; infotheo vs praznik packages

I want to calculate conditional Mutual information in R. As far as I understood, the conditional mutual information takes 3 random variables say X,Y,Z where I(X;Y|Z) means conditional mutual ...
user avatar
  • 33

1
2 3 4 5
8