Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

1
vote
2answers
29 views

Score of importance from feature selection techniques

Can I get the score of importance for each feature in feature selection methos such as Chi2, Information Gain (IG), or Recursive Feature Elimination (RFE)? Or they just provide a list of important ...
5
votes
1answer
95 views

Why not normalizing mutual information with harmonic mean of entropies?

This is a similar question to this one (which has unfortunately no answer yet), although I believe my question is more specific. Let $X$ and $Y$ be two discrete random variables with outcome space, ...
1
vote
0answers
26 views

Why we normalize mutual information by square root of entropy?

Usually NMI(P,T) is expressed as $\frac{MI(P,T)}{\sqrt{H(P)H(T)}}$. However, I don't know the reason why $\sqrt{H(P)H(T)}$ is a maximum value for Mutual information. Is there any proof or explanation?...
0
votes
0answers
18 views

joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
2
votes
0answers
18 views

Bound for type of correlation measure

Assume you have a finite, discrete probability distribution for a joint random variable and such that $P(X=i,Y=j) = p_{i,j}$ for $i \in \{1, \dots, |X|\},j \in \{1, \dots, |Y|\}$. The marginal ...
3
votes
1answer
59 views

Recommended Mutual Information Estimator for Continuous Variable

The mutual information seems to be quite an interesting measure of the relationship between variables. As such I wanted to apply it to investigate the relationship of two continuous variables $X$ and $...
2
votes
1answer
64 views

Alternative definition of Multivariate Mutual Information

The standard mutual information (MI) is given by $$I(X;Y) = H(X) + H(Y) - H(XY)$$ which is the amount of information shared by the two random variables $X$ and $Y$. According to wikipedia article, ...
0
votes
0answers
45 views

Conditional Entropy in the context of Gaussian Processes

I've got a question regarding the conditional entropy of a discrete random variable. According to this paper the conditional entropy of a Gaussian random variable conditioned on a set of variables can ...
0
votes
0answers
25 views

Mutual info between continuous and discrete variables from numerical data

I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed ...
0
votes
0answers
11 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
1
vote
2answers
103 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
0
votes
0answers
16 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
65 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
1
vote
0answers
37 views

Does mutual information depend on the number of data points?

I am playing with mutual information in scikit-learn. ...
1
vote
1answer
50 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
30 views

Cross - entropy for two variables with different prob. distributions

Let us say that we have given two random variables with different prob. distributions: A = [0.1, 0, 0.5, ...] B = [0.3, 0.1, 0.03, ...] What should I do when I want to compute the reformulated cross-...
1
vote
0answers
34 views

Is mutual information symmetric under conditioning?

Cover defines conditional mutual information as I(X; Y|Z) = H(X|Z) - H(X|Y,Z), which I found confusing, as I would have expected I(X; Y|Z) = H(X) - H(X|Y,Z). So I started trying to see which of 1) I(...
2
votes
1answer
36 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
3
votes
2answers
47 views

Infinitesimal independence

Let's say we have two random variables $X$ and $Y$. Is there a name for saying that $X$ and $Y$ are independent only for the values concentrated around a small interval around some $x_0$ and $y_0$ ? ...
0
votes
0answers
39 views

Interpretation of MINE Mutual Estimator function evaluated on individual samples

This paper proposes an estimator for MI over two channels with finite samples. The estimator (eq. 10 in the paper) uses an expression obtained from a parametric NN evaluated over a mixture of joint ...
0
votes
0answers
35 views

Ranking events based on point-wise mutual information with other events

I have three events, x, y, z. I want to rank x, y based on their point-wise mutual information (PMI) values with z; PMI(x,z) and PMI(y,z). I know how to calculate PMI. My problem is that I want to ...
3
votes
1answer
162 views

Which result for normalized mutual information is correct?

I wanted to find the normalized mutual information to validate a clustering algorithm, but I've encountered two different values depending on the library I use. In Python: ...
0
votes
1answer
40 views

Does downsampling decrease the entropy of the data?

Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$. Can we say this downsampling ...
3
votes
2answers
270 views
0
votes
0answers
73 views

On the meaning of mutual information and on how to test the convergence of an estimation

As the result of a Molecular Dynamics simulation, I have the time series of two variables, $X$ and $Y,$ and I am interested in computing the mutual information of these two random variables. I've ...
4
votes
0answers
73 views

Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...
1
vote
0answers
20 views

How Much More Information (as bits) is Gained (if any) from a Ranking vs. Likert Scale Survey of 100 Questions's / 1-5 Scale?

All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a ...
1
vote
1answer
48 views

Where's wrong in my reasoning behind upper bound for reconstruction error?

In the paper Mutual Information Neural Estimation, the authors derive the reconstruction error in BiGAN as $$ \mathcal R=E_{x\sim q(x)}E_{z\sim q(z|x)}\left[-\log p(x|z)\right] $$ where $q(z|x)$ is ...
0
votes
0answers
43 views

Intuitive sense of K-nearest neighbor mutual information estimation

I am using the R package Parmigene to estimate the mutual information (MI) between different proteins. The data is spectral counts, which is nonnegative and mostly zero. I want to know what is ...
4
votes
1answer
362 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
1
vote
0answers
34 views

Downsampling features: how to select the most optimal features to recapitulate clustering [closed]

I've performed single cell analysis in which each gene represents a feature to cluster upon; there are about 20,000 genes expressed across all the cells in the dataset. I use the top 1500 or so ...
0
votes
0answers
22 views

How to calculate PMI when P(x,y) = 0?

I'm trying to calculate PMI using probabilities estimated using binary occurrence counts from a set of documents. In short: $P(x)$ = # documents x appears in / # documents $P(y)$ = # documents y ...
0
votes
0answers
78 views

Is continuous mutual information the correct analogue of the discrete version?

I'm interested in the mutual information of two continuous random variables $X$ and $Y$. Shannon defined differential entropy as $h(X) = -\int p_X(x)\log p_X(x) dx$, where $p_X$ is the probability ...
1
vote
0answers
45 views

How to select value of k in K-NN when using Transfer Entropy?

I was reading about Transfer Entropy and I came across the estimators used to calculate TE, one of them being the Kraskov Estimator: $ T_{X \rightarrow Y} = \frac{p(Y_{n+1}, Y(k)_{n}, X(l)_{n})*log(...
1
vote
0answers
29 views

How does Noise affect the results of Transfer Entropy?

I was reading about Transfer Entropy and came across this package: https://cran.r-project.org/web/packages/TransferEntropy/TransferEntropy.pdf The code in the package: ...
0
votes
1answer
149 views

Meaning and interpretation of Transfer Entropy

I am a first-year undergrad student and I have been reading about Transfer Entropy for my research. Although I understand the math behind I am not really sure what the value means. For example, I run $...
-1
votes
1answer
120 views

Clustering evaluation metric when overclustering is of [closed]

What evaluation metric should be used to measure clustering performance when over-clustering is OK as long as it happens only within ground truth clusters, with no confusion of ground truth clusters. ...
0
votes
0answers
78 views

How can I compute mutual information of continuous random variables from discrete samples?

I have two dependent random variables $X$ and $Y$. $X$ is a continuous random variable and $Y$ is a function of continuous random variables. I know the pdf of $X$ and have a sample from it. I only ...
1
vote
0answers
40 views

Mutual information between continuous $Z$ and $g(Z)$ for differentiable $g$?

I have a continuous random variable $Z$ and a differentiable function $X = g(Z)$. Is the mutual information between $X$ and $Z$ necessarily $\infty$ or 0? Are there any examples of differentiable ...
0
votes
0answers
13 views

Study of Correlation between two variables (bond lenghts) which are strongly affected by a third one (temperature)

In a Molecular Dynamics simulation, I have the time series of a couple of distances between pairs of atoms, d1 (distance between atom A and B), and d2 (distance between other atoms C and D). I want to ...
0
votes
1answer
29 views

To calculate information gain

I want to calculate information gain by using this formula- Here, probabilities are interpreted on an event space of documents (e.g., P(t¯k, ci) denotes the probability that, for a random document x, ...
0
votes
0answers
38 views

Entropy is a measure on distributions, mutual information is a measure on RVs

In a lecture I read this statement, no explanation was given "Unlike entropy, mutual information is a measure on random variables, not on distributions" Can anyone explain what this means? From ...
0
votes
1answer
32 views

Shannon entropy with regards to independent random variables

I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific ...
1
vote
0answers
46 views

Mutual information between multivariate random variables? [duplicate]

I have read that mutual information only works with two random variables, and that for 3 or more RVs there seems to be a variety of different measures (synergy, partial information decomposition, and ...
0
votes
1answer
16 views

When is the mututal information (MI) between two variables (say X and Y) strong enough?

I understand that the mutual information (MI) is a measure of information added to the variable X given the addition of variable Y. I also understand that a MI closer to 0 indicates that the new ...
4
votes
1answer
85 views

covariance to correlation is like mutual information to --?

It is well known that correlation is the normalized covariance, i.e. $\ Cor(X, Y) = Cov(X, Y)/\sqrt{Var(X)Var(Y)}$. These two related measures describe the linear relationships in the data. Is ...
1
vote
0answers
22 views

Conditions Mutual Information and Confounding Effect

Given that conditional mutual information (CMI) I(A,B |C) is the information shared between A, and B given C, does this consider the confounding effect -if any - that C introduces? In other words, ...
1
vote
1answer
19 views

Estimating the mutual information in high dimension when all but one variable are iid

I have a function $f(x_{1},\dots,x_{n})$ where $n$ is large and I would like to estimate the mutual information between the random variable $f(X_{1},\dots,X_{n})$ and the independent and identically ...
1
vote
0answers
474 views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
1
vote
0answers
295 views

Trying to implement the Jensen-Shannon Divergence for Multivariate Gaussians

Given two multivariate Gaussian distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them. ...