Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

384 questions
Filter by
Sorted by
Tagged with
12 views

Mutual Information of nonadjacent nodes in Bayesian Network

How do you compute the mutual information of two non-adjacent nodes in a Bayesian network? In this case, what would $I(D;A)$ be? Would I need to take the conditional probabilities of all intemediate ...
13 views

Hierarchical Clustering Using Mutual Information

I am interested in Hierarchical Clustering Using Mutual Information. Asking the ChatGpt, I got this: ...
1 vote
58 views

Chain rule conditional entropy

A textbook I am reading states that$$H(X,Y)=H(X)+H(Y|X)$$where $H(X,Y)$ is the joint entropy of random variables $X,Y$, $H(X)$ the entropy of $X$, and $H(Y|X)$ is conditional entropy. It then states ...
• 125
1 vote
28 views

Mutual Information decay

Consider $m$ channels indexed by $i$ with $1 \leq i \leq m$. The input alphabets are from the same finite set $\mathcal{X}$. Let $\pi$ denote a probability distribution on $\mathcal{X}$. Define the ...
31 views

K-nearest neighbors to estimate mutual information

I would like to use the mutual_info_regression object from scikit-learn to get a rough idea of how well any individual feature ...
• 1,276
45 views

Does maximizing log likelihood lead to maximizing mutual information?

In supervised learning, we usually maximize log-likelihood, for example, by minimizing cross-entropy loss. Let's say we have data $X$ and their respective labels $Y$. We train a model to output a ...
47 views

76 views

Calculating mutual Information in absence of a token/string over a class

Equation: MI(X,Ci) = ∑P(X,Ci) . log ( P(X,Ci) / P(X)⋅P(Ci) ) where, X is url, t is a url token and Ci is the i-th class will be either spam or ham P(t,C) will be either the frequency that the word &...
1 vote
73 views

Measure the Mutual Information between two variables that is also shared by a third variable

Is there a way to measure how much of the information that two variables share is also shared by a third variable? Say that there are three variables $X$, $Y$, $Z$ and that I need to: Predict $Y$ ...
• 391
62 views

Mutual Information between the mean of normally-distributed random variables and another variable

Given normally-distributed (possibly dependent) random variables $X_1 \dots X_n$, their mean $X=\frac{1}{n} \sum_iX_i$, and another discrete r.v. $Y$, can we relate $MI(X,Y)$ with the individual \$MI(...
• 277
1k views

mutual information between discrete random variables and continuous variables

How does one compute mutual information (MI) between discrete random variables and continuous random variables ? It's easy to compute MI for dataset with only discrete random variables. For dataset ...
• 1,216
1 vote
127 views

Mutual Information larger than smaller one of both entropies?

Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
• 197
1 vote
51 views

Why would I get a non-zero value for mutual information for one variable, say x_m (belonging to (x_1, ..., x_n)), to target y, if x_m is a constant?

I have n 'features', (x_1, ..., x_n) and a target variable y. I use sklearn's mutual information score for feature selection to determine which features are the most important for the target y. I get ...
• 71