Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

277 questions
Filter by
Sorted by
Tagged with
182 views

covariance to correlation is like mutual information to --?

It is well known that correlation is the normalized covariance, i.e. $\ Cor(X, Y) = Cov(X, Y)/\sqrt{Var(X)Var(Y)}$. These two related measures describe the linear relationships in the data. Is ...
38 views

Conditions Mutual Information and Confounding Effect

Given that conditional mutual information (CMI) I(A,B |C) is the information shared between A, and B given C, does this consider the confounding effect -if any - that C introduces? In other words, ...
27 views

Estimating the mutual information in high dimension when all but one variable are iid

I have a function $f(x_{1},\dots,x_{n})$ where $n$ is large and I would like to estimate the mutual information between the random variable $f(X_{1},\dots,X_{n})$ and the independent and identically ...
3k views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
1k views

Trying to implement the Jensen-Shannon Divergence for Multivariate Gaussians

Given two multivariate Gaussian distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them. ...
1k views

Entropy of a function of independent random variables

Suppose I have an operator (function) $f(\cdot)$ which takes three arguments $x,y,z$ all of which are independent random variables, and all of which I have access to the probability mass function (...
80 views

Entropy of a factorised joint distribution

Suppose I have three discrete random variables $X, Y$ and $Z$. Their joint distribution factorises as so: $$P(X,Y,Z) = P(X)P(Y)P(Z)$$ i.e. they are fully independent variables. Now suppose I want ...
785 views

Methods for evaluation of clustering

I have labeled data set (with only 2 classes) and I'm trying different clustering algorithms with different variations of similarity measures (which creates different distance matrixes that I give as ...
60 views

Simplification of delta mutual information formula

I'm calculating the difference between two mutual information's, to see if adding a new parameter is worth. My goal is to simplify the formula which might be a little CPU intensive, but I can't see ...
134 views

Evaluating rare event risk metrics

Suppose there is a rare event that happens on 3-7 days a year, and we are interested to predict days when it happens. We have two metrics, A and B, that both take values on onterval (0, 1) for any ...
53 views

When can a probability or entropy relation converted to an equivalent conditional statement?

For some probability rules such as $$p(y,x) = p(y|x) p(x)$$ there is a similar version that is conditioned on an extra variable, like this case $$p(y,x|z) = p(y|x,z) p(x|z)$$ Also for ...
87 views

Mutual Information from Multiple Sources

The mutual information gain expression is $$H(X) - H(X | Y)$$ If I have a set of data sources, $\mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace$, then I start with the simplest mutual ...
110 views

Information gain calculation using mutual information

I am wondering how to calculate information gain using mutual information. With the help of python's sckit-learn, i have calculated mutual information between two features directly, but there is no ...
4k views

mutual information for feature selection

I'm having difficulty (again) understanding mutual information for feature selection. Here's some R code: ...
76 views

what is the additivity of mutual information?

I've heard that if $x$ and $y$ are independent, then the following inequality holds $I(z;x) + I(z;y) \le I(z;x,y)$ But if $x$ and $y$ are not independent, then how would such inequality change? ...
150 views

A measure of similarity between two different classifiers

This is a purely hypothetical question and doesn't relate to any specific application. Suppose you have two separate linear boundaries that can divide a bunch of data points into either categories A ...
68 views

Bayesian Optimization: How do we compute the maximum increase in information for the next step

If I understand correctly, Bayesian Optimization works by having a prior and choosing the next step by finding the direction that will maximize the mutual information between the prior and posterior. ...
53 views

Why does $\sum_{x,y} p(x,y) log(P(x)P(y))$ decrease as the random variables $X$ and $Y$ become more dependant?

I think this must be so because of how Mutual Information is defined. The formula is $$I = \sum_{x,y} [p(x,y) \log(p(x,y)) - p(x,y)\log(p(x)p(y))]$$ This is apparently $\ge0$. I can understand it ...
2k views

What is the range of information gain ratio?

I am wondering what the value range of information gain ratio is. I guess it is [0,1] but am not too sure about it.
856 views

226 views

Variations in Bayes' theorem in the denominator

I've got a question regarding the different variations of the denominator in Bayes' Theorem. I'm led to believe the denominator of this equation is = P(B) I did some research and read about a rule ...
971 views

How to be absolutely sure that features do have predictive power to predict the labels (without domain knowledge) ? Does Mutual information help?

Im working on a classification problem which has a severe class imbalance (Its more like an anomaly detection at this point since majority class constitutes 97.5 percent of the dataset ) . Ive tried a ...
75 views

Measure Information content of test samples compared to whole data set

Imagine I am limited to a certain amount of training samples to train a classifier. I am wondering if there is a measure to tell how informative one sample (or a collection of a few samples) of the ...
784 views

Minimizing the mutual information

Lets say I have two independent random variables $T$ and $D$ both over finite integers sets $S_T$ and $S_D$ respectively. Also, assume the probability function of $D$ is given to us and is only ...
124 views

Approximate string matching with uncertainty

Background: I have an interesting little problem where I need some sort of metric to compare strings (or list of numbers of whatever), with the added caveat that the base/ground string (the thing ...
437 views

mutual information of two identical signals [duplicate]

library(infotheo) > mutinformation(c(1,2,3),c(1,2,3)) [1] 1.098612 > mutinformation(c(1,1,1),c(1,1,1)) [1] 0 Why is there a difference? I keep reading that ...
28 views

Proof of information-theoretic approach of likelihood calculation of Bayes Nets

I am trying to understand the proof of the scoring function (log likelihood) of a graph $G$ of a Bayes Network given in this slide. Given a Bayes Net $G$, the log likelihood of data $D$ can be ...
Consider random variables $X_i \sim Bern(0.5)$. We know that the conditional entropy $H(X_1,X_2)$ is maximized when $X_2$ is independent of $X_1$, i.e, the joint distribution $\mathbb{P}(X_1=x_1, X_2=... 1answer 213 views What is “class” in mutual information based feature selection? I'm having a little hard time understanding this specific feature selection algorithm. Specifically, I am looking into maximum-relevance-minimum-redundancy method for feature selection. If I have a ... 0answers 153 views how can i calculate the mutual information I have a vector of continuous random variables X and Y. Y has the value of 0 or 1. I want to calculate which random variable from vector X has more information that makes Y to be 1 using mutual ... 0answers 245 views How to calculate mutual information between a phrase and a list of words I am using this paper Finding Semantic Orientation of Reviews Using Unsupervised PMI Algorithm to determine semantic orientation between a phrase and a list of words using pointwize mutual ... 2answers 2k views Jaccard similarity coefficient vs. Point-wise mutual information coefficient Can you explain the difference between the Jaccard similarity coefficient and the pointwise mutual information (PMI) measure? It would be great if you could add a few examples. 1answer 115 views clustering using mutual information as metric I have a set of data that I would like to cluster. Considered some domain features, I think that Mutual Information is a pretty good measure of how much to elements of the dataset are close one to ... 1answer 187 views Calculating mutual information over distance I'm having trouble reproducing this figure measuring mutual information as a function of the distance between symbols in text/music/genome/etc: from https://arxiv.org/pdf/1606.06737v2.pdf ... 0answers 146 views Bias correction term for maximum likelihood estimation of mutual information from joint distributions According to this webpage, the bias correction term when estimating$I(X;Y)$for discrete random variables$X,Y$is$\sim \textrm{df}(X,Y)/N$where$\textrm{df}(X,Y)$is the degrees of freedom of the ... 1answer 396 views Invariance of mutual information (two-dimensional Gaussian) I encountered a putative contradiction. Assume we have two 2-dim. Gaussian variables$z_1 = (x_1, y_1)$and$z_2 = (x_2, y_2)$with all components being independent, normal distributed variables:$x_1,...
I consider a circular complex normal variable $x = x_r + i x_i \sim \mathcal{CN}(0, \sigma^2)$. I know that the PDFs of the magnitude $r$ of that variable and the squared magnitude $s = r^2$ are given ...