Questions tagged [mutual-information]
mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.
388 questions
8
votes
4
answers
554
views
Signal-to-noise ratio in predictive modeling and machine learning
The interesting comments to this question get into how signal-to-noise ratio plays into ability to make predictions. Being more explicit about it, how does signal-to-noise ratio factor into how good ...
- 67k
0
votes
0
answers
18
views
mutual information between two discrete and a continuous variables
I am wondering what is the best way to compute the mutual information between discrete ($a_i$ or $a_j$) and continuous $h_{ik}$ values such as this $I(a_i;a_j|h_{ik})$ where $k\in \{1,2,...,K\}$ and $...
- 117
1
vote
1
answer
82
views
Distance Correlation vs Mutual Information for non-linear relationship between features
I am working on getting non-linear relationships between Amount (dependent variable) and other features(independent variables). So far I have used Distance correlation and Mutual Information. But when ...
- 11
0
votes
1
answer
61
views
What is pointwise about pointwise mutual information (PMI)?
I have just run into Positive Pointwise Mutual Information (PPMI) while learning Natural Language Processing. I don't have a good foundation in statistics. I understand why it is positive because in ...
- 121
0
votes
1
answer
64
views
Understanding Mutual Information as a Measure of Relationship: Why is Mutual Information Different for Perfectly Correlated Deterministic Functions?
Can someone explain why the mutual information (MI) between a1 and a2 is smaller than the MI between ...
- 414
0
votes
0
answers
12
views
How to compute mutual information between a scalar random variable and a vector random variable? [duplicate]
I have a vector random variable $X = (x_1, x_2... x_n)$ where each $x_i$ has a discrete value. I also have a discrete scalar random variable $Y = y$. Is mutual information between the two defined? If ...
- 81
1
vote
0
answers
21
views
Using Crossvalidation to evaluate different Mutual Information Threshholds for Feature Selection
I'm currently trying to determine a mutual information threshold value for feature selection in a classification task. My idea was to set different thresholds based on location parameters e.g. median, ...
1
vote
1
answer
30
views
Why using mutual information is allowed for feature selection if depends on the "scale" of entropies?
It is common to use mutual information as feature selection method. However, I fail to see why this is the case, since the mutual information $I(X, Y)$ depends on both entropies $H(X)$ and $H(Y)$ via ...
- 621
0
votes
0
answers
18
views
Mutual Information of nonadjacent nodes in Bayesian Network
How do you compute the mutual information of two non-adjacent nodes in a Bayesian network?
In this case, what would $I(D;A)$ be? Would I need to take the conditional probabilities of all intemediate ...
- 35
0
votes
0
answers
26
views
Hierarchical Clustering Using Mutual Information
I am interested in Hierarchical Clustering Using Mutual Information. Asking the ChatGpt, I got this:
...
- 292
2
votes
1
answer
107
views
Chain rule conditional entropy
A textbook I am reading states that$$H(X,Y)=H(X)+H(Y|X)$$where $H(X,Y)$ is the joint entropy of random variables $X,Y$, $H(X)$ the entropy of $X$, and $H(Y|X)$ is conditional entropy. It then states ...
- 187
1
vote
0
answers
32
views
Mutual Information decay
Consider $m$ channels indexed by $i$ with $1 \leq i \leq m$. The input alphabets are from the same finite set $\mathcal{X}$. Let $\pi$ denote a probability distribution on $\mathcal{X}$. Define the ...
- 155
2
votes
0
answers
58
views
K-nearest neighbors to estimate mutual information
I would like to use the mutual_info_regression object from scikit-learn to get a rough idea of how well any individual feature ...
- 1,256
0
votes
0
answers
71
views
Does maximizing log likelihood lead to maximizing mutual information?
In supervised learning, we usually maximize log-likelihood, for example, by minimizing cross-entropy loss. Let's say we have data $X$ and their respective labels $Y$. We train a model to output a ...
0
votes
0
answers
51
views
Mutual information with convex combination of dependent variable and random noise
Suppose I have random variables $A, B, C$ where $B$ is statistically-independent of $A$ and $C$ (e.g. it is random noise). Consider scalars $0 < \alpha < \beta < 1$ and random variables $X = (...
- 1
0
votes
1
answer
55
views
Information Value Definition on Siddiqui "Credit Risk Scorecards" Book
I was studying about information value, and saw the definition on "Siddiqui, Credit Risk Scorecards". I'm searchin for some references on why the formula there is that way, and also for the ...
1
vote
0
answers
33
views
Does Mutual Information by the power of 3 for the numerator really exist?
I am currently trying to measure collocational strength using Mutual Information (MI). MI gives an edge for exclusive and infrequent words. As stated by Brezina (2018), in measuring collocational ...
user404350
0
votes
0
answers
7
views
What is the best statistical measurement for collocational analysis?
I am a beginner with statistics and corpus linguistics, so sorry in advance if my explanation have gaps in it.
So I want to perform corpus linguistics collocational strength analysis on a given corpus,...
user404350
1
vote
0
answers
16
views
Analytical expression of mutual information between a multivariate Gaussian and a binary class variable
The goal is to construct settings that allow analytical computations of the mutual information in the form:
$$I(X; Y) = H(X) - H(X | Y)$$
I am wondering if that is ...
- 307
1
vote
0
answers
175
views
Theoretically optimal "loss" function for continuous distributions? [closed]
Classification problems have the benefit of being discrete, so it's easy to calculate how much information your model has. For example, in LLMs your training data has inputs $\mathbf{x}_i$, a list of ...
- 121
0
votes
0
answers
28
views
Query regarding MI and NMI in the context of image fusion
I am exploring Mutual Information (MI) and Normalized Mutual Information (NMI) in the context of image fusion. While reviewing various sources, it's often mentioned that Mutual Information's value ...
- 1
2
votes
1
answer
296
views
Conditional mutual information $I(X;Y|Z)$
If there is a function $Y=f(Z)$, is the following mutual information equal to zero?
\begin{align}
I(X;Y|Z)=0
\end{align}
Intuitively, it is correct. But how can we prove this?
- 227
1
vote
0
answers
55
views
Markov chain data processing inequality
For a Markov chain $X \rightarrow Y \rightarrow Z$, we have the following data processing inequality: $I(Y;X) \ge I(Z;X)$. Now for the Markov chain, $(W,X) \rightarrow Y \rightarrow Z$, can we prove ...
2
votes
1
answer
84
views
What is the Mutual Information for one instance?
I'm computing mutual information for several features where one of my datasets has one instance. One instance is because of a specific filtering criterion I used.
I'm using ...
- 121
1
vote
1
answer
462
views
How to interpret a mutual information value less than 1
For a mutual information of two continuous variables (X and Y), I interpret a value of M.I. = 1 bit (i.e., 2^1 = 2 distinguishable levels), to mean the following:
If I know any given value of X, I ...
- 739
0
votes
1
answer
68
views
Why is the WordPiece algorithm implemented according to the maximum mutual information?
WordPiece is a subword segmentation algorithm in the field of natural language processing. Different from BPE, WordPiece will select a pair with the largest mutual information to merge each time, and ...
- 11
1
vote
0
answers
87
views
Using Copulas to find mutual information
I have two multidimensional datasets $X, Y$ of dimensions $m \times n$. Here $m$ is the successive measurements and $n$ is the data collected during each measurement. We can say each of $m$ are ...
- 127
1
vote
1
answer
168
views
Negative Mutual information python
I calculated mutual information with the PyInform function with
local = True
MI = mutual_info(xs, ys, local=True)
In blue you see MI between the yellow and green line.
Can anyone tell me why MI ...
- 13
0
votes
1
answer
66
views
Jeffreys divergence for a normal bivariate
I am reading the book Information Theory and Statistics of S. Kullback. In page 8 ((4.3) it is shown that the KL divergence between the joint bivariate normal and the product of the corresponding ...
- 11
1
vote
0
answers
101
views
Mutual Information between Gaussians in the limiting, strongly correlated case
In this question it is derived that if X, Y are correlated Gaussian variables, the mutual information between them is given by
$$ I(X;Y) = \log\left(\frac{\det(\Sigma_X)\det(\Sigma_Y)}{\det\Sigma_{XY}}...
0
votes
1
answer
89
views
How is the rate $H(X\mid Y)$ achieved in Slepian-Wolf coding?
Given two (generally correlated) sources $X,Y$, Slepian-Wolf coding is a protocol that shows it's possible to encode them separately, then have $X$ send $Y$ only $n H(X\mid Y)$ bits of information, ...
- 393
0
votes
0
answers
74
views
Can We Use Mutual Information to Determine if a Time Series has a Difference Trend or a Time Trend?
Let's say we have a finite real-valued time series (finite subsequence of a realization of a real-valued stochastic process), $X_t$.
To address my question, we make no initial stationarity assumptions....
- 451
0
votes
1
answer
174
views
Finding trend similarity between two different time series
I am very new to this data analytics field, so bit confused about what to do.
I have 2 time series datasets of two different products that have the same attributes. I want to find trend similarity ...
- 101
1
vote
1
answer
219
views
Mutual information I(X, Y) >= I(f(X), f(Y)) for deterministic f?
I have the intuition that applying a deterministic function to a pair random variables cannot increase their mutual information, because the function can only decrease each of their entropies.
I would ...
- 521
0
votes
0
answers
23
views
What do the authors mean by this, regarding estimation of mutual information
I found the following while reading a paper [1] and got confused:
Replacing $k_{ij}$, $k{i.}$, respectively $k_{.j}$, by $k_{i,j}$, $k_{i.}$ and $k_{.j}$ provides us with estimates of entropy and ...
- 1
1
vote
0
answers
19
views
How do I numerically compute $I(X;CX+Y)$?
Given that $X\sim\text{Bernoulli}(\nu)$ for some $\nu\in(0,1)$, and $Y\sim N(0,1)$ are independent random variables. I want to compute the mutual information $I(X;CX+Y)$, where $C$ is some known non-...
- 229
0
votes
0
answers
28
views
Model marginal and joint distributions from a sample of unkown number of categories
To illustrate the problems imagine I'm drawing labelled spheres from a box. I may or may not know the number of spheres in the box (does it make a difference?)
If I draw 10 spheres from the box and ...
- 521
1
vote
2
answers
542
views
Is mutual information defined for random variables that are vectors?
I know that mutual information is a measure for how similar two random variables are. There is plenty of information available about how to empirically determine mutual information empirically from ...
0
votes
0
answers
26
views
What is the non-asymptotic counterpart of the mutual information?
The mutual information of a joint probability distribution $p(x,y)$ tells us, if we send $n$-letter messages with each letter drawn from the marginal distribution $p_X(x)$, that we can use roughly $2^{...
- 393
1
vote
1
answer
774
views
Use Mutual Information Feature Selection For Categorical Feature
I have a dataset in which there are Features of both float and object type . I want to apply feature selection On this dataset in such a way that fisrt Find Mutual Information Score of all the ...
- 21
3
votes
1
answer
395
views
Mutual Information result in python for Feature selection unexpected result
I've started to study feature selection techniques and i have a situation that I don't understand.
I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are ...
5
votes
1
answer
4k
views
Scikit-learn: mutual info regression
My understanding of the mutual information between two random variables X and Y can be stated formally as follows:
$$I(X ; Y) = H(X) — H(X | Y)$$
Where $I(X; Y)$ is the mutual information for $X$ and $...
- 1,256
1
vote
0
answers
13
views
How to statistically test whether the co-existing of events is more likely after a treatment (paired data)
I have some nominal (binary) data at two time points. The data structure are as follows:
participant
event A
event B
event C
event D
1 Before
TRUE
TRUE
TRUE
FALSE
2 Before
TRUE
FALSE
TRUE
FALSE
3 ...
- 207
0
votes
1
answer
29
views
Quantifying + comparing the effects of conditioning on Shannon entropy
I'm working on a project that compares how predictable/unpredictable individuals' actions are in terms of how they transition between actions. We consider their actions to part of a first-order Markov ...
- 1
2
votes
1
answer
184
views
Maximize Mutual Information between multiple ("overlapping") RVs
I'd like to maximize the sum of Mutual Information between a RV $X$ and $K$ out of $N$ possible RVs $Z_i$.
$$ \max \sum_{i \in K} \text{MI}(X, Z_i) $$
However, when I unfold the sum I get
$$ \sum_{i \...
- 219
2
votes
1
answer
356
views
What is the information plane theorem for an autoencoder neural network?
Slide 8 (about 19 minutes into the video) of the Stanford Seminar - Information Theory of Deep Learning, Naftali Tishby has the following (rather informally stated) theorem.
Theorem (Information ...
- 9,660
2
votes
2
answers
171
views
Surprisal in rankings
I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race.
Intuitively, in a race with 100 people, ...
- 1,163
2
votes
0
answers
41
views
How much do we learn about a random subset? [closed]
Suppose we sample the following two random variables, for some large integer $n$:
Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
- 21
0
votes
0
answers
93
views
Calculating mutual Information in absence of a token/string over a class
Equation:
MI(X,Ci) = ∑P(X,Ci) . log ( P(X,Ci) / P(X)⋅P(Ci) )
where, X is url, t is a url token and Ci is the i-th class will be either spam or ham
P(t,C) will be either the frequency that the word &...
1
vote
1
answer
80
views
Measure the Mutual Information between two variables that is also shared by a third variable
Is there a way to measure how much of the information that two variables share is also shared by a third variable?
Say that there are three variables $X$, $Y$, $Z$ and that I need to:
Predict $Y$ ...
- 391