Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

0
votes
0answers
8 views

Mutual info between discrete and continuous RV with history dependence

I am looking for literature references/measures to compute the mutual information between a continuous variable (time series) and a binary variable (a temporal sequence of 0's/1's). Briefly, a time ...
1
vote
2answers
89 views

Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error. ...
0
votes
0answers
12 views

Simple expression related to Mutual Information

One way to define the mutual information is $I(X;Y) = H(X) - H(X|Y)$ I have found it useful to look the related quantity $?(X;Y=y) = H(X) - H(X|Y=y)$ That is, we look at how much the entropy of $X$...
0
votes
0answers
17 views

Feature selection using information gain numeric features

I am trying to perform feature selection using the information gain criteria i.e. with the information.gain function in the FSelector R package and I am at a loss to what to do with my features that ...
1
vote
0answers
25 views

Does mutual information depend on the number of data points?

I am playing with mutual information in scikit-learn. ...
0
votes
0answers
14 views

How is the upper bound for Normalized Mutual Information determined?

Mutual information between two clusterings $A$ and $B$ can be calculated as: $$MI(A,B)=H(A)+H(B)-H(A,B)$$ In the 10th page of this paper it is stated that $MI(A,B)$ can vary in the range $[0,\min\{H(...
0
votes
0answers
25 views

Cross - entropy for two variables with different prob. distributions

Let us say that we have given two random variables with different prob. distributions: A = [0.1, 0, 0.5, ...] B = [0.3, 0.1, 0.03, ...] What should I do when I want to compute the reformulated cross-...
1
vote
0answers
29 views

Is mutual information symmetric under conditioning?

Cover defines conditional mutual information as I(X; Y|Z) = H(X|Z) - H(X|Y,Z), which I found confusing, as I would have expected I(X; Y|Z) = H(X) - H(X|Y,Z). So I started trying to see which of 1) I(...
2
votes
1answer
25 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
0
votes
0answers
39 views

Interpretation of MINE Mutual Estimator function evaluated on individual samples

This paper proposes an estimator for MI over two channels with finite samples. The estimator (eq. 10 in the paper) uses an expression obtained from a parametric NN evaluated over a mixture of joint ...
0
votes
0answers
29 views

Ranking events based on point-wise mutual information with other events

I have three events, x, y, z. I want to rank x, y based on their point-wise mutual information (PMI) values with z; PMI(x,z) and PMI(y,z). I know how to calculate PMI. My problem is that I want to ...
3
votes
1answer
81 views

Which result for normalized mutual information is correct?

I wanted to find the normalized mutual information to validate a clustering algorithm, but I've encountered two different values depending on the library I use. In Python: ...
0
votes
1answer
35 views

Does downsampling decrease the entropy of the data?

Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$. Can we say this downsampling ...
3
votes
2answers
169 views
0
votes
0answers
48 views

On the meaning of mutual information and on how to test the convergence of an estimation

As the result of a Molecular Dynamics simulation, I have the time series of two variables, $X$ and $Y,$ and I am interested in computing the mutual information of these two random variables. I've ...
4
votes
0answers
71 views

Maximizing the information gain on a Gaussian RV with a noisy comparison question

The question Let $X \sim \mathcal{N}(0,1)$ be a random variable denoting the location of a target on the real line. $Y_a$ be a binary random variable encoding the (noisy) answer to the question: "is ...
1
vote
0answers
20 views

How Much More Information (as bits) is Gained (if any) from a Ranking vs. Likert Scale Survey of 100 Questions's / 1-5 Scale?

All other sources of noise controlled, I, know that asking users to rank their preference on a collection of items from "best" (5) to "least" (1) in lieu of having them rate each one individually on a ...
0
votes
0answers
34 views

Where's wrong in my reasoning behind upper bound for reconstruction error?

In the paper Mutual Information Neural Estimation, the authors derive the reconstruction error in BiGAN as $$ \mathcal R=E_{x\sim q(x)}E_{z\sim q(z|x)}\left[-\log p(x|z)\right] $$ where $q(z|x)$ is ...
0
votes
0answers
29 views

Intuitive sense of K-nearest neighbor mutual information estimation

I am using the R package Parmigene to estimate the mutual information (MI) between different proteins. The data is spectral counts, which is nonnegative and mostly zero. I want to know what is ...
3
votes
1answer
211 views

Is stochastic gradient descent biased?

In the paper Mutual Information Neural Estimation, the authors derive the following gradient for the network $$ \nabla_\theta\mathcal V(\theta)=\mathbb E\left[\nabla_\theta T_\theta\right]-{\mathbb E\...
1
vote
0answers
29 views

Downsampling features: how to select the most optimal features to recapitulate clustering [closed]

I've performed single cell analysis in which each gene represents a feature to cluster upon; there are about 20,000 genes expressed across all the cells in the dataset. I use the top 1500 or so ...
0
votes
0answers
22 views

How to calculate PMI when P(x,y) = 0?

I'm trying to calculate PMI using probabilities estimated using binary occurrence counts from a set of documents. In short: $P(x)$ = # documents x appears in / # documents $P(y)$ = # documents y ...
0
votes
0answers
53 views

Is continuous mutual information the correct analogue of the discrete version?

I'm interested in the mutual information of two continuous random variables $X$ and $Y$. Shannon defined differential entropy as $h(X) = -\int p_X(x)\log p_X(x) dx$, where $p_X$ is the probability ...
1
vote
0answers
43 views

How to select value of k in K-NN when using Transfer Entropy?

I was reading about Transfer Entropy and I came across the estimators used to calculate TE, one of them being the Kraskov Estimator: $ T_{X \rightarrow Y} = \frac{p(Y_{n+1}, Y(k)_{n}, X(l)_{n})*log(...
1
vote
0answers
26 views

How does Noise affect the results of Transfer Entropy?

I was reading about Transfer Entropy and came across this package: https://cran.r-project.org/web/packages/TransferEntropy/TransferEntropy.pdf The code in the package: ...
0
votes
1answer
114 views

Meaning and interpretation of Transfer Entropy

I am a first-year undergrad student and I have been reading about Transfer Entropy for my research. Although I understand the math behind I am not really sure what the value means. For example, I run $...
-1
votes
1answer
95 views

Clustering evaluation metric when overclustering is of [closed]

What evaluation metric should be used to measure clustering performance when over-clustering is OK as long as it happens only within ground truth clusters, with no confusion of ground truth clusters. ...
0
votes
0answers
68 views

How can I compute mutual information of continuous random variables from discrete samples?

I have two dependent random variables $X$ and $Y$. $X$ is a continuous random variable and $Y$ is a function of continuous random variables. I know the pdf of $X$ and have a sample from it. I only ...
1
vote
0answers
34 views

Mutual information between continuous $Z$ and $g(Z)$ for differentiable $g$?

I have a continuous random variable $Z$ and a differentiable function $X = g(Z)$. Is the mutual information between $X$ and $Z$ necessarily $\infty$ or 0? Are there any examples of differentiable ...
0
votes
0answers
13 views

Study of Correlation between two variables (bond lenghts) which are strongly affected by a third one (temperature)

In a Molecular Dynamics simulation, I have the time series of a couple of distances between pairs of atoms, d1 (distance between atom A and B), and d2 (distance between other atoms C and D). I want to ...
0
votes
1answer
29 views

To calculate information gain

I want to calculate information gain by using this formula- Here, probabilities are interpreted on an event space of documents (e.g., P(t¯k, ci) denotes the probability that, for a random document x, ...
0
votes
0answers
34 views

Entropy is a measure on distributions, mutual information is a measure on RVs

In a lecture I read this statement, no explanation was given "Unlike entropy, mutual information is a measure on random variables, not on distributions" Can anyone explain what this means? From ...
0
votes
1answer
29 views

Shannon entropy with regards to independent random variables

I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific ...
1
vote
0answers
44 views

Mutual information between multivariate random variables? [duplicate]

I have read that mutual information only works with two random variables, and that for 3 or more RVs there seems to be a variety of different measures (synergy, partial information decomposition, and ...
0
votes
1answer
15 views

When is the mututal information (MI) between two variables (say X and Y) strong enough?

I understand that the mutual information (MI) is a measure of information added to the variable X given the addition of variable Y. I also understand that a MI closer to 0 indicates that the new ...
3
votes
1answer
68 views

covariance to correlation is like mutual information to --?

It is well known that correlation is the normalized covariance, i.e. $\ Cor(X, Y) = Cov(X, Y)/\sqrt{Var(X)Var(Y)}$. These two related measures describe the linear relationships in the data. Is ...
1
vote
0answers
20 views

Conditions Mutual Information and Confounding Effect

Given that conditional mutual information (CMI) I(A,B |C) is the information shared between A, and B given C, does this consider the confounding effect -if any - that C introduces? In other words, ...
1
vote
1answer
18 views

Estimating the mutual information in high dimension when all but one variable are iid

I have a function $f(x_{1},\dots,x_{n})$ where $n$ is large and I would like to estimate the mutual information between the random variable $f(X_{1},\dots,X_{n})$ and the independent and identically ...
1
vote
0answers
376 views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
1
vote
0answers
237 views

Trying to implement the Jensen-Shannon Divergence for Multivariate Gaussians

Given two multivariate Gaussian distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them. ...
0
votes
0answers
27 views

Mutual Information when taking union of symbols

Let $\mathcal{X} = \{ \mathcal{X}_1, \dots, \mathcal{X}_J\}$ and $\mathcal{Y} = \{ \mathcal{Y}_1, \dots, \mathcal{Y}_K\}$ be two finite sets of symbols and let $p(\mathcal{X}_j, \mathcal{Y}_k)$ be ...
2
votes
2answers
335 views

Entropy of a function of independent random variables

Suppose I have an operator (function) $f(\cdot)$ which takes three arguments $x,y,z$ all of which are independent random variables, and all of which I have access to the probability mass function (...
0
votes
0answers
34 views

Entropy of a factorised joint distribution

Suppose I have three discrete random variables $X, Y$ and $Z$. Their joint distribution factorises as so: $$ P(X,Y,Z) = P(X)P(Y)P(Z) $$ i.e. they are fully independent variables. Now suppose I want ...
-1
votes
2answers
186 views

Methods for evaluation of clustering

I have labeled data set (with only 2 classes) and I'm trying different clustering algorithms with different variations of similarity measures (which creates different distance matrixes that I give as ...
0
votes
1answer
42 views

Simplification of delta mutual information formula

I'm calculating the difference between two mutual information's, to see if adding a new parameter is worth. My goal is to simplify the formula which might be a little CPU intensive, but I can't see ...
2
votes
1answer
62 views

Evaluating rare event risk metrics

Suppose there is a rare event that happens on 3-7 days a year, and we are interested to predict days when it happens. We have two metrics, A and B, that both take values on onterval (0, 1) for any ...
1
vote
0answers
52 views

When can a probability or entropy relation converted to an equivalent conditional statement?

For some probability rules such as $$ p(y,x) = p(y|x) p(x) $$ there is a similar version that is conditioned on an extra variable, like this case $$ p(y,x|z) = p(y|x,z) p(x|z) $$ Also for ...
0
votes
0answers
33 views

Mutual Information for more than two variables

Is there any formulas for mutual information for more than two variables, I know this one of two variables which is: $$I(X; Y) = \sum_{y \in Y}\sum_{x \in X}p(x,y)log\Bigg(\frac{p(x,y)}{p(x)p(y)}\...
0
votes
0answers
48 views

Mutual Information from Multiple Sources

The mutual information gain expression is $$ H(X) - H(X | Y) $$ If I have a set of data sources, $ \mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace $, then I start with the simplest mutual ...
2
votes
0answers
46 views

Information gain calculation using mutual information

I am wondering how to calculate information gain using mutual information. With the help of python's sckit-learn, i have calculated mutual information between two features directly, but there is no ...