Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
0
votes
0answers
55 views

How to choose sample size from probability density for computing mutual information based on continuous variables

I need to compute mutual information gain based two continuous variables $X$ and $Y$ $I(X|Y) = \int_X\int_Y p_{x.y}(x,y) \log(\frac{p_{x.y}(x,y)}{p_{x}(x)p_{y}(y)})$. I have used Kernel Density ...
0
votes
0answers
14 views

How would one calculate the mutual information between two random vectors?

I have a vector of three random variables $\mathbf{X} = \{x_1, x_2,x_3\}$, and a vector of five random variables $\mathbf{Y} = \{ y_1, y_2, ..., y_5\}$. The two vectors are not independent. I would ...
0
votes
1answer
36 views

If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
0
votes
1answer
14 views

Mutual Information between multi-dimensional and single dimensional Variables

I would like to estimate MI between two variables X and Y of shape (nXd) and ...
0
votes
1answer
33 views

Information Bottleneck principle of Deep Learning model implementation

I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other ...
0
votes
0answers
32 views

Why isn't “mutual information” instead called “mutual entropy”, and “pointwise mutual information” instead called “mutual information”?

Unless I misunderstand something, the following points are true: The entropy of a variable is the average information that you get from it with each trial. The mutual information between two ...
0
votes
1answer
34 views

If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
0
votes
0answers
15 views

Difference of mutual informations / Kullback-Leibler divergences for dependent arbitrary- and Gaussian random variables with similar second moments

Let $(Y_1, Y_2)$ be arbitrarily jointly distributed random variables, and let $(Y_{1,G}, Y_{1,G})$ be jointly distributed Gaussian random variables with the same mean and second moments as those of $(...
0
votes
0answers
33 views

Choosing appropriate statistical tests

I am trying to run stats on medical treatment data and would like your guys and girls help Goal: Comparing two trends of annual percentages for statistical significance. Is one decreasing due to the ...
0
votes
0answers
8 views

Finding most related variables independent of variable type

Problem I am analysing a dataset containing variables of different types: continuous, ordinal and categorical. To prioritise in which order to analyse the variables, I would like to evaluate the ...
0
votes
0answers
14 views

Pointwise Mutual Information using spacy or just detailed explaination

So, I have been trying to play around with NLP recently and decided to work on a project involving Emotional Analysis. I have been following this particular research, http://www.cse.yorku.ca/~aan/...
1
vote
0answers
24 views

What is the purpose of finding the Maximum Spanning Tree?

I'm referring to Chow-Liu algorithm in Bayesian network structure learning. We first construct a Mutual Information Graph, and from that we find the Maximum Spanning Tree. But, once we got the tree, ...
0
votes
1answer
37 views

How to normalize mutual information between to real-valued random variables?

How can I normalize mutual information between to real-valued random variables using Python or R? ...
0
votes
0answers
40 views
0
votes
0answers
20 views

Can a Jeffreys prior be used as an Information maximizing distribution if Information is defined using differential entropy?

I know that a Jeffreys prior is the information maximizing distribution for the statistical channel. However, I want to know if I define mutual information as $$I(x;y)=h(x)-h(x|y)$$ where $h(.)$ is ...
0
votes
1answer
24 views

How to calculate mutual information between a feature and target variable?

Mutual information measures how much information the distribution of one variable provides about the distribution of another variable. In my case, I have samples of a feature variable $X \in \mathbb{...
1
vote
2answers
75 views

Score of importance from feature selection techniques

Can I get the score of importance for each feature in feature selection methos such as Chi2, Information Gain (IG), or Recursive Feature Elimination (RFE)? Or they just provide a list of important ...
0
votes
0answers
16 views

Is there an archive of closed-form mutual information among the “famous” distributions?

I'm looking for a document or compilation table of closed-form mutual information as a function of their parameters, for known distributions such as normal, gamma, Poisson distributions. At least, I ...
5
votes
1answer
116 views

Why not normalizing mutual information with harmonic mean of entropies?

This is a similar question to this one (which has unfortunately no answer yet), although I believe my question is more specific. Let $X$ and $Y$ be two discrete random variables with outcome space, ...
1
vote
0answers
76 views

Why we normalize mutual information by square root of entropy?

Usually NMI(P,T) is expressed as $\frac{MI(P,T)}{\sqrt{H(P)H(T)}}$. However, I don't know the reason why $\sqrt{H(P)H(T)}$ is a maximum value for Mutual information. Is there any proof or explanation?...
0
votes
0answers
36 views

joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
2
votes
0answers
19 views

Bound for type of correlation measure

Assume you have a finite, discrete probability distribution for a joint random variable and such that $P(X=i,Y=j) = p_{i,j}$ for $i \in \{1, \dots, |X|\},j \in \{1, \dots, |Y|\}$. The marginal ...
3
votes
1answer
65 views

Recommended Mutual Information Estimator for Continuous Variable

The mutual information seems to be quite an interesting measure of the relationship between variables. As such I wanted to apply it to investigate the relationship of two continuous variables $X$ and $...
2
votes
1answer
76 views

Alternative definition of Multivariate Mutual Information

The standard mutual information (MI) is given by $$I(X;Y) = H(X) + H(Y) - H(XY)$$ which is the amount of information shared by the two random variables $X$ and $Y$. According to wikipedia article, ...
0
votes
0answers
97 views

Conditional Entropy in the context of Gaussian Processes

I've got a question regarding the conditional entropy of a discrete random variable. According to this paper the conditional entropy of a Gaussian random variable conditioned on a set of variables can ...
0
votes
0answers
50 views

Mutual info between continuous and discrete variables from numerical data

I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed ...
0
votes
0answers
17 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
2
votes
2answers
154 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
0
votes
0answers
19 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
150 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
1
vote
0answers
56 views

Does mutual information depend on the number of data points?

I am playing with mutual information in scikit-learn. ...
2
votes
1answer
116 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
47 views

Cross - entropy for two variables with different prob. distributions

Let us say that we have given two random variables with different prob. distributions: A = [0.1, 0, 0.5, ...] B = [0.3, 0.1, 0.03, ...] What should I do when I want to compute the reformulated cross-...
1
vote
0answers
48 views

Is mutual information symmetric under conditioning?

Cover defines conditional mutual information as I(X; Y|Z) = H(X|Z) - H(X|Y,Z), which I found confusing, as I would have expected I(X; Y|Z) = H(X) - H(X|Y,Z). So I started trying to see which of 1) I(...
2
votes
1answer
55 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
5
votes
2answers
66 views

Infinitesimal independence

Let's say we have two random variables $X$ and $Y$. Is there a name for saying that $X$ and $Y$ are independent only for the values concentrated around a small interval around some $x_0$ and $y_0$ ? ...
0
votes
0answers
45 views

Ranking events based on point-wise mutual information with other events

I have three events, x, y, z. I want to rank x, y based on their point-wise mutual information (PMI) values with z; PMI(x,z) and PMI(y,z). I know how to calculate PMI. My problem is that I want to ...
3
votes
1answer
260 views

Which result for normalized mutual information is correct?

I wanted to find the normalized mutual information to validate a clustering algorithm, but I've encountered two different values depending on the library I use. In Python: ...
0
votes
1answer
51 views

Does downsampling decrease the entropy of the data?

Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$. Can we say this downsampling ...
4
votes
2answers
505 views

Correcting Kullback-Leibler divergence for size of datasets

We have the following implementation of KLD: ...
4
votes
0answers
76 views

Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...
1
vote
0answers
21 views

How Much More Information (as bits) is Gained (if any) from a Ranking vs. Likert Scale Survey of 100 Questions's / 1-5 Scale?

All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a ...
1
vote
1answer
66 views

Where's wrong in my reasoning behind upper bound for reconstruction error?

In the paper Mutual Information Neural Estimation, the authors derive the reconstruction error in BiGAN as $$ \mathcal R=E_{x\sim q(x)}E_{z\sim q(z|x)}\left[-\log p(x|z)\right] $$ where $q(z|x)$ is ...
0
votes
0answers
91 views

Intuitive sense of K-nearest neighbor mutual information estimation

I am using the R package Parmigene to estimate the mutual information (MI) between different proteins. The data is spectral counts, which is nonnegative and mostly zero. I want to know what is ...
4
votes
1answer
637 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
1
vote
0answers
60 views

Downsampling features: how to select the most optimal features to recapitulate clustering [closed]

I've performed single cell analysis in which each gene represents a feature to cluster upon; there are about 20,000 genes expressed across all the cells in the dataset. I use the top 1500 or so ...
0
votes
0answers
23 views

How to calculate PMI when P(x,y) = 0?

I'm trying to calculate PMI using probabilities estimated using binary occurrence counts from a set of documents. In short: $P(x)$ = # documents x appears in / # documents $P(y)$ = # documents y ...
1
vote
0answers
137 views

Is continuous mutual information the correct analogue of the discrete version?

I'm interested in the mutual information of two continuous random variables $X$ and $Y$. Shannon defined differential entropy as $h(X) = -\int p_X(x)\log p_X(x) dx$, where $p_X$ is the probability ...
1
vote
0answers
50 views

How to select value of k in K-NN when using Transfer Entropy?

I was reading about Transfer Entropy and I came across the estimators used to calculate TE, one of them being the Kraskov Estimator: $ T_{X \rightarrow Y} = \frac{p(Y_{n+1}, Y(k)_{n}, X(l)_{n})*log(...
1
vote
0answers
35 views

How does Noise affect the results of Transfer Entropy?

I was reading about Transfer Entropy and came across this package: https://cran.r-project.org/web/packages/TransferEntropy/TransferEntropy.pdf The code in the package: ...