Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

0
votes
0answers
10 views

Mutual Information between layer's activation and class label

I want to calculate Information gain of particular layer's activation with respect to class label, which is something quite similar to, ...
0
votes
0answers
11 views

Why isn't “mutual information” instead called “mutual entropy”, and “pointwise mutual information” instead called “mutual information”?

Unless I misunderstand something: The entropy of a variable is the average information that you get from it with each trial. The mutual information between two variables is the average amount by ...
0
votes
1answer
32 views

If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
0
votes
0answers
14 views

Difference of mutual informations / Kullback-Leibler divergences for dependent arbitrary- and Gaussian random variables with similar second moments

Let $(Y_1, Y_2)$ be arbitrarily jointly distributed random variables, and let $(Y_{1,G}, Y_{1,G})$ be jointly distributed Gaussian random variables with the same mean and second moments as those of $(...
0
votes
0answers
29 views

Choosing appropriate statistical tests

I am trying to run stats on medical treatment data and would like your guys and girls help Goal: Comparing two trends of annual percentages for statistical significance. Is one decreasing due to the ...
0
votes
0answers
8 views

Finding most related variables independent of variable type

Problem I am analysing a dataset containing variables of different types: continuous, ordinal and categorical. To prioritise in which order to analyse the variables, I would like to evaluate the ...
0
votes
0answers
9 views

Pointwise Mutual Information using spacy or just detailed explaination

So, I have been trying to play around with NLP recently and decided to work on a project involving Emotional Analysis. I have been following this particular research, http://www.cse.yorku.ca/~aan/...
0
votes
0answers
8 views

Optimzation of Mutual Information among datasets

Consider $3$ datasets : $d_1$, $d_2$ and $d_3$. We calculate mutual information $M_1$ between $d_1$ and $d_3$ and $M_2$ between $d_2$, $d_3$. Given that the data in $d_3$ can be changed so that both ...
1
vote
0answers
21 views

What is the purpose of finding the Maximum Spanning Tree?

I'm referring to Chow-Liu algorithm in Bayesian network structure learning. We first construct a Mutual Information Graph, and from that we find the Maximum Spanning Tree. But, once we got the tree, ...
0
votes
1answer
26 views

How to normalize mutual information between to real-valued random variables?

How can I normalize mutual information between to real-valued random variables using Python or R? ...
0
votes
0answers
19 views

Can a Jeffreys prior be used as an Information maximizing distribution if Information is defined using differential entropy?

I know that a Jeffreys prior is the information maximizing distribution for the statistical channel. However, I want to know if I define mutual information as $$I(x;y)=h(x)-h(x|y)$$ where $h(.)$ is ...
0
votes
1answer
19 views

How to calculate mutual information between a feature and target variable?

Mutual information measures how much information the distribution of one variable provides about the distribution of another variable. In my case, I have samples of a feature variable $X \in \mathbb{...
1
vote
2answers
48 views

Score of importance from feature selection techniques

Can I get the score of importance for each feature in feature selection methos such as Chi2, Information Gain (IG), or Recursive Feature Elimination (RFE)? Or they just provide a list of important ...
0
votes
0answers
15 views

Is there an archive of closed-form mutual information among the “famous” distributions?

I'm looking for a document or compilation table of closed-form mutual information as a function of their parameters, for known distributions such as normal, gamma, Poisson distributions. At least, I ...
5
votes
1answer
105 views

Why not normalizing mutual information with harmonic mean of entropies?

This is a similar question to this one (which has unfortunately no answer yet), although I believe my question is more specific. Let $X$ and $Y$ be two discrete random variables with outcome space, ...
1
vote
0answers
49 views

Why we normalize mutual information by square root of entropy?

Usually NMI(P,T) is expressed as $\frac{MI(P,T)}{\sqrt{H(P)H(T)}}$. However, I don't know the reason why $\sqrt{H(P)H(T)}$ is a maximum value for Mutual information. Is there any proof or explanation?...
0
votes
0answers
29 views

joint entropy and mutual information

If we have joint entropy of 10 bits between distribution A and B, while mutual information is 2 bits. Can we say there is 0.4% of useful communication between two distributions? Would this be proper ...
2
votes
0answers
19 views

Bound for type of correlation measure

Assume you have a finite, discrete probability distribution for a joint random variable and such that $P(X=i,Y=j) = p_{i,j}$ for $i \in \{1, \dots, |X|\},j \in \{1, \dots, |Y|\}$. The marginal ...
3
votes
1answer
63 views

Recommended Mutual Information Estimator for Continuous Variable

The mutual information seems to be quite an interesting measure of the relationship between variables. As such I wanted to apply it to investigate the relationship of two continuous variables $X$ and $...
2
votes
1answer
72 views

Alternative definition of Multivariate Mutual Information

The standard mutual information (MI) is given by $$I(X;Y) = H(X) + H(Y) - H(XY)$$ which is the amount of information shared by the two random variables $X$ and $Y$. According to wikipedia article, ...
0
votes
0answers
71 views

Conditional Entropy in the context of Gaussian Processes

I've got a question regarding the conditional entropy of a discrete random variable. According to this paper the conditional entropy of a Gaussian random variable conditioned on a set of variables can ...
0
votes
0answers
30 views

Mutual info between continuous and discrete variables from numerical data

I am looking for references/measures to estimate the mutual information between a continuous (C) and discrete (D) variable, given a real-world (i.e. finite sample) data set. C is uniformly distributed ...
0
votes
0answers
14 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
1
vote
2answers
117 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
0
votes
0answers
18 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
114 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
1
vote
0answers
46 views

Does mutual information depend on the number of data points?

I am playing with mutual information in scikit-learn. ...
1
vote
1answer
97 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
35 views

Cross - entropy for two variables with different prob. distributions

Let us say that we have given two random variables with different prob. distributions: A = [0.1, 0, 0.5, ...] B = [0.3, 0.1, 0.03, ...] What should I do when I want to compute the reformulated cross-...
1
vote
0answers
43 views

Is mutual information symmetric under conditioning?

Cover defines conditional mutual information as I(X; Y|Z) = H(X|Z) - H(X|Y,Z), which I found confusing, as I would have expected I(X; Y|Z) = H(X) - H(X|Y,Z). So I started trying to see which of 1) I(...
2
votes
1answer
45 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
5
votes
2answers
64 views

Infinitesimal independence

Let's say we have two random variables $X$ and $Y$. Is there a name for saying that $X$ and $Y$ are independent only for the values concentrated around a small interval around some $x_0$ and $y_0$ ? ...
0
votes
0answers
37 views

Ranking events based on point-wise mutual information with other events

I have three events, x, y, z. I want to rank x, y based on their point-wise mutual information (PMI) values with z; PMI(x,z) and PMI(y,z). I know how to calculate PMI. My problem is that I want to ...
3
votes
1answer
209 views

Which result for normalized mutual information is correct?

I wanted to find the normalized mutual information to validate a clustering algorithm, but I've encountered two different values depending on the library I use. In Python: ...
0
votes
1answer
48 views

Does downsampling decrease the entropy of the data?

Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$. Can we say this downsampling ...
3
votes
2answers
378 views
4
votes
0answers
76 views

Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...
1
vote
0answers
20 views

How Much More Information (as bits) is Gained (if any) from a Ranking vs. Likert Scale Survey of 100 Questions's / 1-5 Scale?

All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a ...
1
vote
1answer
58 views

Where's wrong in my reasoning behind upper bound for reconstruction error?

In the paper Mutual Information Neural Estimation, the authors derive the reconstruction error in BiGAN as $$ \mathcal R=E_{x\sim q(x)}E_{z\sim q(z|x)}\left[-\log p(x|z)\right] $$ where $q(z|x)$ is ...
0
votes
0answers
78 views

Intuitive sense of K-nearest neighbor mutual information estimation

I am using the R package Parmigene to estimate the mutual information (MI) between different proteins. The data is spectral counts, which is nonnegative and mostly zero. I want to know what is ...
4
votes
1answer
534 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
1
vote
0answers
51 views

Downsampling features: how to select the most optimal features to recapitulate clustering [closed]

I've performed single cell analysis in which each gene represents a feature to cluster upon; there are about 20,000 genes expressed across all the cells in the dataset. I use the top 1500 or so ...
0
votes
0answers
23 views

How to calculate PMI when P(x,y) = 0?

I'm trying to calculate PMI using probabilities estimated using binary occurrence counts from a set of documents. In short: $P(x)$ = # documents x appears in / # documents $P(y)$ = # documents y ...
1
vote
0answers
110 views

Is continuous mutual information the correct analogue of the discrete version?

I'm interested in the mutual information of two continuous random variables $X$ and $Y$. Shannon defined differential entropy as $h(X) = -\int p_X(x)\log p_X(x) dx$, where $p_X$ is the probability ...
1
vote
0answers
47 views

How to select value of k in K-NN when using Transfer Entropy?

I was reading about Transfer Entropy and I came across the estimators used to calculate TE, one of them being the Kraskov Estimator: $ T_{X \rightarrow Y} = \frac{p(Y_{n+1}, Y(k)_{n}, X(l)_{n})*log(...
1
vote
0answers
32 views

How does Noise affect the results of Transfer Entropy?

I was reading about Transfer Entropy and came across this package: https://cran.r-project.org/web/packages/TransferEntropy/TransferEntropy.pdf The code in the package: ...
1
vote
1answer
180 views

Meaning and interpretation of Transfer Entropy

I am a first-year undergrad student and I have been reading about Transfer Entropy for my research. Although I understand the math behind I am not really sure what the value means. For example, I run $...
-1
votes
1answer
138 views

Clustering evaluation metric when overclustering is of [closed]

What evaluation metric should be used to measure clustering performance when over-clustering is OK as long as it happens only within ground truth clusters, with no confusion of ground truth clusters. ...
0
votes
0answers
94 views

How can I compute mutual information of continuous random variables from discrete samples?

I have two dependent random variables $X$ and $Y$. $X$ is a continuous random variable and $Y$ is a function of continuous random variables. I know the pdf of $X$ and have a sample from it. I only ...