# Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

277 questions
Filter by
Sorted by
Tagged with
182 views

### covariance to correlation is like mutual information to --?

It is well known that correlation is the normalized covariance, i.e. $\ Cor(X, Y) = Cov(X, Y)/\sqrt{Var(X)Var(Y)}$. These two related measures describe the linear relationships in the data. Is ...
38 views

### Conditions Mutual Information and Confounding Effect

Given that conditional mutual information (CMI) I(A,B |C) is the information shared between A, and B given C, does this consider the confounding effect -if any - that C introduces? In other words, ...
27 views

### Estimating the mutual information in high dimension when all but one variable are iid

I have a function $f(x_{1},\dots,x_{n})$ where $n$ is large and I would like to estimate the mutual information between the random variable $f(X_{1},\dots,X_{n})$ and the independent and identically ...
3k views

### Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
1k views

### Trying to implement the Jensen-Shannon Divergence for Multivariate Gaussians

Given two multivariate Gaussian distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them. ...
1k views

### Entropy of a function of independent random variables

Suppose I have an operator (function) $f(\cdot)$ which takes three arguments $x,y,z$ all of which are independent random variables, and all of which I have access to the probability mass function (...
80 views

### Entropy of a factorised joint distribution

Suppose I have three discrete random variables $X, Y$ and $Z$. Their joint distribution factorises as so: $$P(X,Y,Z) = P(X)P(Y)P(Z)$$ i.e. they are fully independent variables. Now suppose I want ...
785 views

### Methods for evaluation of clustering

I have labeled data set (with only 2 classes) and I'm trying different clustering algorithms with different variations of similarity measures (which creates different distance matrixes that I give as ...
60 views

### Simplification of delta mutual information formula

I'm calculating the difference between two mutual information's, to see if adding a new parameter is worth. My goal is to simplify the formula which might be a little CPU intensive, but I can't see ...
134 views

### Evaluating rare event risk metrics

Suppose there is a rare event that happens on 3-7 days a year, and we are interested to predict days when it happens. We have two metrics, A and B, that both take values on onterval (0, 1) for any ...
53 views

### When can a probability or entropy relation converted to an equivalent conditional statement?

For some probability rules such as $$p(y,x) = p(y|x) p(x)$$ there is a similar version that is conditioned on an extra variable, like this case $$p(y,x|z) = p(y|x,z) p(x|z)$$ Also for ...
87 views

### Mutual Information from Multiple Sources

The mutual information gain expression is $$H(X) - H(X | Y)$$ If I have a set of data sources, $\mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace$, then I start with the simplest mutual ...
110 views

### Information gain calculation using mutual information

I am wondering how to calculate information gain using mutual information. With the help of python's sckit-learn, i have calculated mutual information between two features directly, but there is no ...
4k views

### mutual information for feature selection

I'm having difficulty (again) understanding mutual information for feature selection. Here's some R code: ...
76 views

### what is the additivity of mutual information?

I've heard that if $x$ and $y$ are independent, then the following inequality holds $I(z;x) + I(z;y) \le I(z;x,y)$ But if $x$ and $y$ are not independent, then how would such inequality change? ...
150 views

### A measure of similarity between two different classifiers

This is a purely hypothetical question and doesn't relate to any specific application. Suppose you have two separate linear boundaries that can divide a bunch of data points into either categories A ...
68 views

### Bayesian Optimization: How do we compute the maximum increase in information for the next step

If I understand correctly, Bayesian Optimization works by having a prior and choosing the next step by finding the direction that will maximize the mutual information between the prior and posterior. ...
53 views

### Why does $\sum_{x,y} p(x,y) log(P(x)P(y))$ decrease as the random variables $X$ and $Y$ become more dependant?

I think this must be so because of how Mutual Information is defined. The formula is $$I = \sum_{x,y} [p(x,y) \log(p(x,y)) - p(x,y)\log(p(x)p(y))]$$ This is apparently $\ge0$. I can understand it ...
2k views

### What is the range of information gain ratio?

I am wondering what the value range of information gain ratio is. I guess it is [0,1] but am not too sure about it.
856 views

226 views

### Variations in Bayes' theorem in the denominator

I've got a question regarding the different variations of the denominator in Bayes' Theorem. I'm led to believe the denominator of this equation is = P(B) I did some research and read about a rule ...
971 views

### How to be absolutely sure that features do have predictive power to predict the labels (without domain knowledge) ? Does Mutual information help?

Im working on a classification problem which has a severe class imbalance (Its more like an anomaly detection at this point since majority class constitutes 97.5 percent of the dataset ) . Ive tried a ...
75 views

### Measure Information content of test samples compared to whole data set

Imagine I am limited to a certain amount of training samples to train a classifier. I am wondering if there is a measure to tell how informative one sample (or a collection of a few samples) of the ...
784 views

### Minimizing the mutual information

Lets say I have two independent random variables $T$ and $D$ both over finite integers sets $S_T$ and $S_D$ respectively. Also, assume the probability function of $D$ is given to us and is only ...
124 views

### Approximate string matching with uncertainty

Background: I have an interesting little problem where I need some sort of metric to compare strings (or list of numbers of whatever), with the added caveat that the base/ground string (the thing ...
437 views

### mutual information of two identical signals [duplicate]

library(infotheo) > mutinformation(c(1,2,3),c(1,2,3))  1.098612 > mutinformation(c(1,1,1),c(1,1,1))  0 Why is there a difference? I keep reading that ...
28 views

### Proof of information-theoretic approach of likelihood calculation of Bayes Nets

I am trying to understand the proof of the scoring function (log likelihood) of a graph $G$ of a Bayes Network given in this slide. Given a Bayes Net $G$, the log likelihood of data $D$ can be ...
Consider random variables $X_i \sim Bern(0.5)$. We know that the conditional entropy $H(X_1,X_2)$ is maximized when $X_2$ is independent of $X_1$, i.e, the joint distribution $\mathbb{P}(X_1=x_1, X_2=... 1answer 213 views ### What is “class” in mutual information based feature selection? I'm having a little hard time understanding this specific feature selection algorithm. Specifically, I am looking into maximum-relevance-minimum-redundancy method for feature selection. If I have a ... 0answers 153 views ### how can i calculate the mutual information I have a vector of continuous random variables X and Y. Y has the value of 0 or 1. I want to calculate which random variable from vector X has more information that makes Y to be 1 using mutual ... 0answers 245 views ### How to calculate mutual information between a phrase and a list of words I am using this paper Finding Semantic Orientation of Reviews Using Unsupervised PMI Algorithm to determine semantic orientation between a phrase and a list of words using pointwize mutual ... 2answers 2k views ### Jaccard similarity coefficient vs. Point-wise mutual information coefficient Can you explain the difference between the Jaccard similarity coefficient and the pointwise mutual information (PMI) measure? It would be great if you could add a few examples. 1answer 115 views ### clustering using mutual information as metric I have a set of data that I would like to cluster. Considered some domain features, I think that Mutual Information is a pretty good measure of how much to elements of the dataset are close one to ... 1answer 187 views ### Calculating mutual information over distance I'm having trouble reproducing this figure measuring mutual information as a function of the distance between symbols in text/music/genome/etc: from https://arxiv.org/pdf/1606.06737v2.pdf ... 0answers 146 views ### Bias correction term for maximum likelihood estimation of mutual information from joint distributions According to this webpage, the bias correction term when estimating$I(X;Y)$for discrete random variables$X,Y$is$\sim \textrm{df}(X,Y)/N$where$\textrm{df}(X,Y)$is the degrees of freedom of the ... 1answer 396 views ### Invariance of mutual information (two-dimensional Gaussian) I encountered a putative contradiction. Assume we have two 2-dim. Gaussian variables$z_1 = (x_1, y_1)$and$z_2 = (x_2, y_2)$with all components being independent, normal distributed variables:$x_1,...
I consider a circular complex normal variable $x = x_r + i x_i \sim \mathcal{CN}(0, \sigma^2)$. I know that the PDFs of the magnitude $r$ of that variable and the squared magnitude $s = r^2$ are given ...