Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
4
votes
1answer
182 views

covariance to correlation is like mutual information to --?

It is well known that correlation is the normalized covariance, i.e. $\ Cor(X, Y) = Cov(X, Y)/\sqrt{Var(X)Var(Y)}$. These two related measures describe the linear relationships in the data. Is ...
1
vote
0answers
38 views

Conditions Mutual Information and Confounding Effect

Given that conditional mutual information (CMI) I(A,B |C) is the information shared between A, and B given C, does this consider the confounding effect -if any - that C introduces? In other words, ...
1
vote
1answer
27 views

Estimating the mutual information in high dimension when all but one variable are iid

I have a function $f(x_{1},\dots,x_{n})$ where $n$ is large and I would like to estimate the mutual information between the random variable $f(X_{1},\dots,X_{n})$ and the independent and identically ...
1
vote
1answer
3k views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
2
votes
1answer
1k views

Trying to implement the Jensen-Shannon Divergence for Multivariate Gaussians

Given two multivariate Gaussian distributions $P \equiv \mathcal{N}(\mu_p, \Sigma_p)$ and $Q \equiv \mathcal{N}(\mu_q, \Sigma_q)$, I am trying to calculate the Jensen-Shannon divergence between them. ...
2
votes
2answers
1k views

Entropy of a function of independent random variables

Suppose I have an operator (function) $f(\cdot)$ which takes three arguments $x,y,z$ all of which are independent random variables, and all of which I have access to the probability mass function (...
0
votes
0answers
80 views

Entropy of a factorised joint distribution

Suppose I have three discrete random variables $X, Y$ and $Z$. Their joint distribution factorises as so: $$ P(X,Y,Z) = P(X)P(Y)P(Z) $$ i.e. they are fully independent variables. Now suppose I want ...
-1
votes
2answers
785 views

Methods for evaluation of clustering

I have labeled data set (with only 2 classes) and I'm trying different clustering algorithms with different variations of similarity measures (which creates different distance matrixes that I give as ...
0
votes
1answer
60 views

Simplification of delta mutual information formula

I'm calculating the difference between two mutual information's, to see if adding a new parameter is worth. My goal is to simplify the formula which might be a little CPU intensive, but I can't see ...
2
votes
1answer
134 views

Evaluating rare event risk metrics

Suppose there is a rare event that happens on 3-7 days a year, and we are interested to predict days when it happens. We have two metrics, A and B, that both take values on onterval (0, 1) for any ...
1
vote
0answers
53 views

When can a probability or entropy relation converted to an equivalent conditional statement?

For some probability rules such as $$ p(y,x) = p(y|x) p(x) $$ there is a similar version that is conditioned on an extra variable, like this case $$ p(y,x|z) = p(y|x,z) p(x|z) $$ Also for ...
0
votes
0answers
87 views

Mutual Information from Multiple Sources

The mutual information gain expression is $$ H(X) - H(X | Y) $$ If I have a set of data sources, $ \mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace $, then I start with the simplest mutual ...
2
votes
0answers
110 views

Information gain calculation using mutual information

I am wondering how to calculate information gain using mutual information. With the help of python's sckit-learn, i have calculated mutual information between two features directly, but there is no ...
8
votes
1answer
4k views

mutual information for feature selection

I'm having difficulty (again) understanding mutual information for feature selection. Here's some R code: ...
1
vote
0answers
76 views

what is the additivity of mutual information?

I've heard that if $x$ and $y$ are independent, then the following inequality holds $ I(z;x) + I(z;y) \le I(z;x,y)$ But if $x$ and $y$ are not independent, then how would such inequality change? ...
0
votes
0answers
150 views

A measure of similarity between two different classifiers

This is a purely hypothetical question and doesn't relate to any specific application. Suppose you have two separate linear boundaries that can divide a bunch of data points into either categories A ...
3
votes
1answer
68 views

Bayesian Optimization: How do we compute the maximum increase in information for the next step

If I understand correctly, Bayesian Optimization works by having a prior and choosing the next step by finding the direction that will maximize the mutual information between the prior and posterior. ...
3
votes
1answer
53 views

Why does $\sum_{x,y} p(x,y) log(P(x)P(y))$ decrease as the random variables $X$ and $Y$ become more dependant?

I think this must be so because of how Mutual Information is defined. The formula is $$ I = \sum_{x,y} [p(x,y) \log(p(x,y)) - p(x,y)\log(p(x)p(y))] $$ This is apparently $\ge0$. I can understand it ...
4
votes
1answer
2k views

What is the range of information gain ratio?

I am wondering what the value range of information gain ratio is. I guess it is [0,1] but am not too sure about it.
3
votes
0answers
856 views

How to correctly re-formulate mutual information between time series?

I am trying to understand the re-formulation of mutual information between time series presented in Galka et al. They note "If the data is given as a pair of time series $x_t$ and $y_t$, $t = 1, . ....
1
vote
0answers
465 views

how to measure mutual information in deep neural network

I'm considering how to measure mutual information between layers in deep neural network. For example, in a MNIST dataset, with few layers of network. I simply flatten each layer to 1d array and ...
1
vote
1answer
37 views

What's the best approach to compute 'average' for this case?

I have a list of words, with two statistics computed for each entry, i.e. Word Freq MI beautiful girl 2310 12.07 girl gift 50 14.9 2310 is ...
1
vote
1answer
198 views

Integrating pointwise mutual information

let's say that I have two continuous events (random variables) $x$ and $y$. Then what is the integration of the pointwise mutual information between these two events with respect to $x$? $\int$$pmi(...
5
votes
1answer
28k views

How to correctly compute mutual information (Python Example)? [closed]

I am trying to compute mutual information for 2 vectors. I made a general function that recognizes if the data is categorical or continuous. It's really difficult to find simple examples of this ...
0
votes
1answer
63 views

Are A and D mutually exclusive?

If P(A) = 0.2, P(B) = 0.2, P(C) = 0.45, P(D) = 0.15 Then does this mean that A and D are mutually exclusive?
0
votes
0answers
165 views

Logistic regression: precision of p(Y)

There are many usual methods to measure the predictive power of logistic regression (or any method predicting probabilities such as probit regression). Some of them are inspired from R-squared for ...
1
vote
0answers
46 views

Stochastic block model and Graphical channel

While reading about community detection and stochastic block model I came up with the article "Community detection and stochastic block models: recent developments" by Emmanuel Abbe. I saw there a ...
1
vote
1answer
158 views

illogical result mutual information

I have two time series and I am studying the mutual information between them in different parts of the year (I have calculated the mutual information in sliding windows). In january, the mutual ...
2
votes
0answers
69 views

Causal Inference Between Linearly Related Measures Using Solomonoff Induction Approximation

Solomonoff Induction is considered the gold standard for machine learning because it can learn causal structure with the minimum error. However its chief component, the Kolmogorov program ...
2
votes
1answer
246 views

Are there any implementations of bayesian optimization with mutual information as acquisition function?

I'm looking for packages of bayesian optimization with mutual information as acquisition function. There are many implementations of bayesian optimization in some programming languages like python, R,...
2
votes
1answer
80 views

Mutual information for markov chain identity qeustion

given that: $$A-B-C-D \space\space markov \space chain$$ We know that: $$ I(A;C|D) = H(C|D)-H(C|A,D)$$ but does it also say that: $$ H(C|A,D)=H(C|D) \space \space from \space the \space markovian?$$ ...
1
vote
1answer
460 views

Self information of random variable

I've got a problem analytically obtaining the mutual information of a random variable with itself. I have a set X of 7 balls: 3 red 2 blue 2 green Therefore p(x) {3/7,2/7,2/7} I know ...
12
votes
2answers
811 views

Mutual Information as probability

Could the mutual information over the joint entropy: $$ 0 \leq \frac{I(X,Y)}{H(X,Y)} \leq 1$$ be defined as:"The probability of conveying a piece of information from X to Y"? I am sorry for being ...
0
votes
1answer
226 views

Variations in Bayes' theorem in the denominator

I've got a question regarding the different variations of the denominator in Bayes' Theorem. I'm led to believe the denominator of this equation is = P(B) I did some research and read about a rule ...
5
votes
1answer
971 views

How to be absolutely sure that features do have predictive power to predict the labels (without domain knowledge) ? Does Mutual information help?

Im working on a classification problem which has a severe class imbalance (Its more like an anomaly detection at this point since majority class constitutes 97.5 percent of the dataset ) . Ive tried a ...
1
vote
0answers
75 views

Measure Information content of test samples compared to whole data set

Imagine I am limited to a certain amount of training samples to train a classifier. I am wondering if there is a measure to tell how informative one sample (or a collection of a few samples) of the ...
4
votes
1answer
784 views

Minimizing the mutual information

Lets say I have two independent random variables $T$ and $D$ both over finite integers sets $S_T$ and $S_D$ respectively. Also, assume the probability function of $D$ is given to us and is only ...
1
vote
0answers
124 views

Approximate string matching with uncertainty

Background: I have an interesting little problem where I need some sort of metric to compare strings (or list of numbers of whatever), with the added caveat that the base/ground string (the thing ...
1
vote
1answer
437 views

mutual information of two identical signals [duplicate]

library(infotheo) > mutinformation(c(1,2,3),c(1,2,3)) [1] 1.098612 > mutinformation(c(1,1,1),c(1,1,1)) [1] 0 Why is there a difference? I keep reading that ...
1
vote
0answers
28 views

Proof of information-theoretic approach of likelihood calculation of Bayes Nets

I am trying to understand the proof of the scoring function (log likelihood) of a graph $G$ of a Bayes Network given in this slide. Given a Bayes Net $G$, the log likelihood of data $D$ can be ...
2
votes
0answers
717 views

Maximizing Mutual Information

Consider random variables $X_i \sim Bern(0.5)$. We know that the conditional entropy $H(X_1,X_2)$ is maximized when $X_2$ is independent of $X_1$, i.e, the joint distribution $\mathbb{P}(X_1=x_1, X_2=...
2
votes
1answer
213 views

What is “class” in mutual information based feature selection?

I'm having a little hard time understanding this specific feature selection algorithm. Specifically, I am looking into maximum-relevance-minimum-redundancy method for feature selection. If I have a ...
1
vote
0answers
153 views

how can i calculate the mutual information

I have a vector of continuous random variables X and Y. Y has the value of 0 or 1. I want to calculate which random variable from vector X has more information that makes Y to be 1 using mutual ...
1
vote
0answers
245 views

How to calculate mutual information between a phrase and a list of words

I am using this paper Finding Semantic Orientation of Reviews Using Unsupervised PMI Algorithm to determine semantic orientation between a phrase and a list of words using pointwize mutual ...
5
votes
2answers
2k views

Jaccard similarity coefficient vs. Point-wise mutual information coefficient

Can you explain the difference between the Jaccard similarity coefficient and the pointwise mutual information (PMI) measure? It would be great if you could add a few examples.
1
vote
1answer
115 views

clustering using mutual information as metric

I have a set of data that I would like to cluster. Considered some domain features, I think that Mutual Information is a pretty good measure of how much to elements of the dataset are close one to ...
2
votes
1answer
187 views

Calculating mutual information over distance

I'm having trouble reproducing this figure measuring mutual information as a function of the distance between symbols in text/music/genome/etc: from https://arxiv.org/pdf/1606.06737v2.pdf ...
1
vote
0answers
146 views

Bias correction term for maximum likelihood estimation of mutual information from joint distributions

According to this webpage, the bias correction term when estimating $I(X;Y)$ for discrete random variables $X,Y$ is $\sim \textrm{df}(X,Y)/N$ where $\textrm{df}(X,Y)$ is the degrees of freedom of the ...
3
votes
1answer
396 views

Invariance of mutual information (two-dimensional Gaussian)

I encountered a putative contradiction. Assume we have two 2-dim. Gaussian variables $z_1 = (x_1, y_1)$ and $z_2 = (x_2, y_2)$ with all components being independent, normal distributed variables: $x_1,...
1
vote
0answers
55 views

Why are the entropies of magnitude and squared magnitude of Gaussian/normal vector different?

I consider a circular complex normal variable $x = x_r + i x_i \sim \mathcal{CN}(0, \sigma^2)$. I know that the PDFs of the magnitude $r$ of that variable and the squared magnitude $s = r^2$ are given ...