The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

measure of the difference between variation of two time series of probability distributions

I'm looking at a series of particle density probabilities of proteins floating on a cell, these particles move around and also blink, so these density probabilities differ a bit from one time to ...
0
votes
0answers
9 views

Proof help for multivariate mutual information as a sum of entropies

I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies. $J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(...
0
votes
0answers
42 views

Who to attribute information gain to?

I am writing a paper where I examine information gain specifically with regards to feature selection and am wondering what the proper reference should be. I have looked all over and I can't find a ...
1
vote
0answers
28 views

Which has more mutual information with a multivariate Gaussian: its first principal component, or its first factor?

I have a $k$-dimensional Gaussian random variable $X\sim\mathcal{N}(0, \Sigma_X)$. What I want is a 1-dimensional scalar r.v. $Y\sim\mathcal{N}(0,1)$ that is jointly Gaussian with $X$ while maximizing ...
1
vote
0answers
55 views

How to show an alternate data processing inequality concerning KL divergence between conditionals?

Suppose $(Y,X) \sim F \in \mathcal{P(\mathbb{R^d})}$. Consider an arbitrary transformation $f$ that acts on $X$. My intuition is that the following should be a result in information theory: $$ \...
0
votes
0answers
15 views

Finding PCA-like directions in feature space that maximise sensitivity to a target variable

I have a fairly large space of feature variables in which I want to build a predictor for a target variable. My input dataset for training the predictor are sampled from the space using a mix of log ...
3
votes
2answers
56 views

Using conditional mutual information with measurements

I want to compute the mutual information between two discrete random variables $x_1,x_2$ whose PMFs are affected by measurements resp. $z_1,z_2$ (one measurement per variable) in a known way. Based on ...
0
votes
1answer
37 views

Need help understanding Toru Tsujishita' theorem on Triple Information

I struggling to understand point (ii) of Toru Tsujishita's theorem (here) on Triple- (Interaction- or Co-) Information What is meant with the maps $\varphi_j$ and $\varphi_k$? A biunivocal ...
2
votes
1answer
69 views

Confusion about proof in “Representation Learning with Contrastive Predictive Coding”

In the Appendix A.1 of the paper "Representation Learning with Contrastive Predictive Coding", the author prove $\log N-\mathcal L_N$ is the lower bound of mutual information between $x_{t+k}$ and $...
0
votes
0answers
27 views

Finding related variables

I have read this and this question, but non of them answers my question properly. I am not very familiar with concepts of stat, so please bear with me. I have 2 types of variables, type A (circles) ...
0
votes
0answers
61 views

How to choose sample size from probability density for computing mutual information based on continuous variables

I need to compute mutual information gain based two continuous variables $X$ and $Y$ $I(X|Y) = \int_X\int_Y p_{x.y}(x,y) \log(\frac{p_{x.y}(x,y)}{p_{x}(x)p_{y}(y)})$. I have used Kernel Density ...
0
votes
0answers
22 views

How would one calculate the mutual information between two random vectors?

I have a vector of three random variables $\mathbf{X} = \{x_1, x_2,x_3\}$, and a vector of five random variables $\mathbf{Y} = \{ y_1, y_2, ..., y_5\}$. The two vectors are not independent. I would ...
0
votes
1answer
42 views

If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
1
vote
0answers
128 views

Non-negativity of interaction information for special trivariate case

Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies $$ p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ), $$ for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
0
votes
1answer
14 views

Mutual Information between multi-dimensional and single dimensional Variables

I would like to estimate MI between two variables X and Y of shape (nXd) and ...
0
votes
1answer
49 views

Information Bottleneck principle of Deep Learning model implementation

I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other ...
0
votes
0answers
37 views

Why isn't “mutual information” instead called “mutual entropy”, and “pointwise mutual information” instead called “mutual information”?

Unless I misunderstand something, the following points are true: The entropy of a variable is the average information that you get from it with each trial. The mutual information between two ...
0
votes
1answer
35 views

If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
0
votes
0answers
15 views

Difference of mutual informations / Kullback-Leibler divergences for dependent arbitrary- and Gaussian random variables with similar second moments

Let $(Y_1, Y_2)$ be arbitrarily jointly distributed random variables, and let $(Y_{1,G}, Y_{1,G})$ be jointly distributed Gaussian random variables with the same mean and second moments as those of $(...
0
votes
0answers
42 views

Choosing appropriate statistical tests

I am trying to run stats on medical treatment data and would like your guys and girls help Goal: Comparing two trends of annual percentages for statistical significance. Is one decreasing due to the ...
0
votes
0answers
12 views

Finding most related variables independent of variable type

Problem I am analysing a dataset containing variables of different types: continuous, ordinal and categorical. To prioritise in which order to analyse the variables, I would like to evaluate the ...
0
votes
0answers
21 views

Pointwise Mutual Information using spacy or just detailed explaination

So, I have been trying to play around with NLP recently and decided to work on a project involving Emotional Analysis. I have been following this particular research, http://www.cse.yorku.ca/~aan/...
0
votes
1answer
86 views

How to normalize mutual information between to real-valued random variables?

How can I normalize mutual information between to real-valued random variables using Python or R? ...
1
vote
1answer
69 views
0
votes
0answers
22 views

Can a Jeffreys prior be used as an Information maximizing distribution if Information is defined using differential entropy?

I know that a Jeffreys prior is the information maximizing distribution for the statistical channel. However, I want to know if I define mutual information as $$I(x;y)=h(x)-h(x|y)$$ where $h(.)$ is ...
0
votes
1answer
31 views

How to calculate mutual information between a feature and target variable?

Mutual information measures how much information the distribution of one variable provides about the distribution of another variable. In my case, I have samples of a feature variable $X \in \mathbb{...
1
vote
2answers
131 views

Score of importance from feature selection techniques

Can I get the score of importance for each feature in feature selection methos such as Chi2, Information Gain (IG), or Recursive Feature Elimination (RFE)? Or they just provide a list of important ...
0
votes
0answers
16 views

Is there an archive of closed-form mutual information among the “famous” distributions?

I'm looking for a document or compilation table of closed-form mutual information as a function of their parameters, for known distributions such as normal, gamma, Poisson distributions. At least, I ...
5
votes
1answer
124 views

Why not normalizing mutual information with harmonic mean of entropies?

This is a similar question to this one (which has unfortunately no answer yet), although I believe my question is more specific. Let $X$ and $Y$ be two discrete random variables with outcome space, ...
1
vote
0answers
96 views

Why we normalize mutual information by square root of entropy?

Usually NMI(P,T) is expressed as $\frac{MI(P,T)}{\sqrt{H(P)H(T)}}$. However, I don't know the reason why $\sqrt{H(P)H(T)}$ is a maximum value for Mutual information. Is there any proof or explanation?...
0
votes
0answers
42 views

joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
2
votes
0answers
22 views

Bound for type of correlation measure

Assume you have a finite, discrete probability distribution for a joint random variable and such that $P(X=i,Y=j) = p_{i,j}$ for $i \in \{1, \dots, |X|\},j \in \{1, \dots, |Y|\}$. The marginal ...
3
votes
1answer
71 views

Recommended Mutual Information Estimator for Continuous Variable

The mutual information seems to be quite an interesting measure of the relationship between variables. As such I wanted to apply it to investigate the relationship of two continuous variables $X$ and $...
2
votes
1answer
90 views

Alternative definition of Multivariate Mutual Information

The standard mutual information (MI) is given by $$I(X;Y) = H(X) + H(Y) - H(XY)$$ which is the amount of information shared by the two random variables $X$ and $Y$. According to wikipedia article, ...
0
votes
0answers
143 views

Conditional Entropy in the context of Gaussian Processes

I've got a question regarding the conditional entropy of a discrete random variable. According to this paper the conditional entropy of a Gaussian random variable conditioned on a set of variables can ...
0
votes
0answers
80 views

Mutual info between continuous and discrete variables from numerical data

I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed ...
0
votes
0answers
18 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
2
votes
2answers
185 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
0
votes
0answers
20 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
176 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
1
vote
0answers
84 views

Does mutual information depend on the number of data points?

I am playing with mutual information in scikit-learn. ...
2
votes
1answer
152 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
71 views

Cross - entropy for two variables with different prob. distributions

Let us say that we have given two random variables with different prob. distributions: A = [0.1, 0, 0.5, ...] B = [0.3, 0.1, 0.03, ...] What should I do when I want to compute the reformulated cross-...
1
vote
0answers
50 views

Is mutual information symmetric under conditioning?

Cover defines conditional mutual information as I(X; Y|Z) = H(X|Z) - H(X|Y,Z), which I found confusing, as I would have expected I(X; Y|Z) = H(X) - H(X|Y,Z). So I started trying to see which of 1) I(...
2
votes
1answer
69 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
5
votes
2answers
72 views

Infinitesimal independence

Let's say we have two random variables $X$ and $Y$. Is there a name for saying that $X$ and $Y$ are independent only for the values concentrated around a small interval around some $x_0$ and $y_0$ ? ...
3
votes
1answer
358 views

Which result for normalized mutual information is correct?

I wanted to find the normalized mutual information to validate a clustering algorithm, but I've encountered two different values depending on the library I use. In Python: ...
0
votes
1answer
59 views

Does downsampling decrease the entropy of the data?

Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$. Can we say this downsampling ...
4
votes
2answers
688 views

Correcting Kullback-Leibler divergence for size of datasets

We have the following implementation of KLD: ...
4
votes
0answers
80 views

Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...