Questions tagged [mutual-information]
mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.
383
questions
1
vote
0
answers
12
views
K-nearest neighbors to estimate mutual information
I would like to use the mutual_info_regression object from scikit-learn to get a rough idea of how well any individual feature ...
0
votes
0
answers
27
views
Does maximizing log likelihood lead to maximizing mutual information?
In supervised learning, we usually maximize log-likelihood, for example, by minimizing cross-entropy loss. Let's say we have data $X$ and their respective labels $Y$. We train a model to output a ...
0
votes
0
answers
30
views
Mutual information with convex combination of dependent variable and random noise
Suppose I have random variables $A, B, C$ where $B$ is statistically-independent of $A$ and $C$ (e.g. it is random noise). Consider scalars $0 < \alpha < \beta < 1$ and random variables $X = (...
0
votes
0
answers
18
views
Information Value Definition on Siddiqui "Credit Risk Scorecards" Book
I was studying about information value, and saw the definition on "Siddiqui, Credit Risk Scorecards". I'm searchin for some references on why the formula there is that way, and also for the ...
1
vote
0
answers
27
views
Does Mutual Information by the power of 3 for the numerator really exist?
I am currently trying to measure collocational strength using Mutual Information (MI). MI gives an edge for exclusive and infrequent words. As stated by Brezina (2018), in measuring collocational ...
0
votes
0
answers
6
views
What is the best statistical measurement for collocational analysis?
I am a beginner with statistics and corpus linguistics, so sorry in advance if my explanation have gaps in it.
So I want to perform corpus linguistics collocational strength analysis on a given corpus,...
1
vote
0
answers
10
views
Analytical expression of mutual information between a multivariate Gaussian and a binary class variable
The goal is to construct settings that allow analytical computations of the mutual information in the form:
$$I(X; Y) = H(X) - H(X | Y)$$
I am wondering if that is ...
1
vote
0
answers
89
views
Theoretically optimal "loss" function for continuous distributions? [closed]
Classification problems have the benefit of being discrete, so it's easy to calculate how much information your model has. For example, in LLMs your training data has inputs $\mathbf{x}_i$, a list of ...
0
votes
0
answers
25
views
Query regarding MI and NMI in the context of image fusion
I am exploring Mutual Information (MI) and Normalized Mutual Information (NMI) in the context of image fusion. While reviewing various sources, it's often mentioned that Mutual Information's value ...
0
votes
0
answers
17
views
Mutual information between a vector of values and a statistic computed on the vector
I understand that the mutual information between two vectors X, Y from two distributions gives a measure of uncertainty of X knowing information about Y. But is it possible to measure mutual ...
2
votes
1
answer
45
views
Conditional mutual information $I(X;Y|Z)$
If there is a function $Y=f(Z)$, is the following mutual information equal to zero?
\begin{align}
I(X;Y|Z)=0
\end{align}
Intuitively, it is correct. But how can we prove this?
0
votes
0
answers
23
views
a continuous analog for adjusted mutual information
I'm interested in comparing a set of electoral redistricting plans to, which can be conceived as partitions of an area.
A measure like mutual information looks attractive.
Let $U_i$ be a district from ...
1
vote
0
answers
48
views
Markov chain data processing inequality
For a Markov chain $X \rightarrow Y \rightarrow Z$, we have the following data processing inequality: $I(Y;X) \ge I(Z;X)$. Now for the Markov chain, $(W,X) \rightarrow Y \rightarrow Z$, can we prove ...
2
votes
1
answer
48
views
What is the Mutual Information for one instance?
I'm computing mutual information for several features where one of my datasets has one instance. One instance is because of a specific filtering criterion I used.
I'm using ...
0
votes
1
answer
186
views
How to interpret a mutual information value less than 1
For a mutual information of two continuous variables (X and Y), I interpret a value of M.I. = 1 bit (i.e., 2^1 = 2 distinguishable levels), to mean the following:
If I know any given value of X, I ...
0
votes
1
answer
38
views
Why is the WordPiece algorithm implemented according to the maximum mutual information?
WordPiece is a subword segmentation algorithm in the field of natural language processing. Different from BPE, WordPiece will select a pair with the largest mutual information to merge each time, and ...
1
vote
0
answers
60
views
Using Copulas to find mutual information
I have two multidimensional datasets $X, Y$ of dimensions $m \times n$. Here $m$ is the successive measurements and $n$ is the data collected during each measurement. We can say each of $m$ are ...
0
votes
0
answers
48
views
Measuring the epistemic uncertainty mutual information
I possess training data denoted as $D$, a given test input denoted as $x$, and a model that is parameterized by a random variable $\theta$. This model is a neural network containing dropout layers, ...
1
vote
1
answer
97
views
Negative Mutual information python
I calculated mutual information with the PyInform function with
local = True
MI = mutual_info(xs, ys, local=True)
In blue you see MI between the yellow and green line.
Can anyone tell me why MI ...
0
votes
1
answer
44
views
Jeffreys divergence for a normal bivariate
I am reading the book Information Theory and Statistics of S. Kullback. In page 8 ((4.3) it is shown that the KL divergence between the joint bivariate normal and the product of the corresponding ...
1
vote
0
answers
44
views
Mutual Information between Gaussians in the limiting, strongly correlated case
In this question it is derived that if X, Y are correlated Gaussian variables, the mutual information between them is given by
$$ I(X;Y) = \log\left(\frac{\det(\Sigma_X)\det(\Sigma_Y)}{\det\Sigma_{XY}}...
0
votes
1
answer
72
views
How is the rate $H(X\mid Y)$ achieved in Slepian-Wolf coding?
Given two (generally correlated) sources $X,Y$, Slepian-Wolf coding is a protocol that shows it's possible to encode them separately, then have $X$ send $Y$ only $n H(X\mid Y)$ bits of information, ...
0
votes
0
answers
23
views
Pointwise mutual information correction for negative bias
I am studying the point wise mutual information and I have understood that it has some limitations in NPL like the bias towards negative values (it tends to assign high scores to low-frequency events)....
0
votes
0
answers
54
views
Can We Use Mutual Information to Determine if a Time Series has a Difference Trend or a Time Trend?
Let's say we have a finite real-valued time series (finite subsequence of a realization of a real-valued stochastic process), $X_t$.
To address my question, we make no initial stationarity assumptions....
0
votes
0
answers
22
views
Is there a natural extension of the ACF/PACF to more general measures of dependency?
Assume that we have a time series that we want to model as a stationary real-valued stochastic process: $$X_t , t \in \{0, 1, \dots \}.$$
Two complementary measures of linear dependence between the ...
0
votes
1
answer
122
views
Finding trend similarity between two different time series
I am very new to this data analytics field, so bit confused about what to do.
I have 2 time series datasets of two different products that have the same attributes. I want to find trend similarity ...
1
vote
1
answer
108
views
Mutual information I(X, Y) >= I(f(X), f(Y)) for deterministic f?
I have the intuition that applying a deterministic function to a pair random variables cannot increase their mutual information, because the function can only decrease each of their entropies.
I would ...
0
votes
0
answers
22
views
What do the authors mean by this, regarding estimation of mutual information
I found the following while reading a paper [1] and got confused:
Replacing $k_{ij}$, $k{i.}$, respectively $k_{.j}$, by $k_{i,j}$, $k_{i.}$ and $k_{.j}$ provides us with estimates of entropy and ...
1
vote
0
answers
19
views
How do I numerically compute $I(X;CX+Y)$?
Given that $X\sim\text{Bernoulli}(\nu)$ for some $\nu\in(0,1)$, and $Y\sim N(0,1)$ are independent random variables. I want to compute the mutual information $I(X;CX+Y)$, where $C$ is some known non-...
0
votes
0
answers
36
views
Trying to understand the derivation in the Information Bottleneck Method
I'm trying to understand the proofs on the The information bottleneck method paper by Tishby, Pereira and Bialek without luck. In particular, the second term in the functional derivative of the ...
0
votes
0
answers
25
views
Model marginal and joint distributions from a sample of unkown number of categories
To illustrate the problems imagine I'm drawing labelled spheres from a box. I may or may not know the number of spheres in the box (does it make a difference?)
If I draw 10 spheres from the box and ...
1
vote
2
answers
317
views
Is mutual information defined for random variables that are vectors?
I know that mutual information is a measure for how similar two random variables are. There is plenty of information available about how to empirically determine mutual information empirically from ...
0
votes
0
answers
24
views
What is the non-asymptotic counterpart of the mutual information?
The mutual information of a joint probability distribution $p(x,y)$ tells us, if we send $n$-letter messages with each letter drawn from the marginal distribution $p_X(x)$, that we can use roughly $2^{...
1
vote
1
answer
530
views
Use Mutual Information Feature Selection For Categorical Feature
I have a dataset in which there are Features of both float and object type . I want to apply feature selection On this dataset in such a way that fisrt Find Mutual Information Score of all the ...
3
votes
1
answer
297
views
Mutual Information result in python for Feature selection unexpected result
I've started to study feature selection techniques and i have a situation that I don't understand.
I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are ...
0
votes
0
answers
276
views
How to compute mutual information between two samples when some features are discrete and some are continuous?
If we need to compute mutual information between two features (columns) where each feature can only be either continuous or discrete, it's easy to compute mutual information between them using ...
5
votes
1
answer
3k
views
Scikit-learn: mutual info regression
My understanding of the mutual information between two random variables X and Y can be stated formally as follows:
$$I(X ; Y) = H(X) — H(X | Y)$$
Where $I(X; Y)$ is the mutual information for $X$ and $...
1
vote
0
answers
12
views
How to statistically test whether the co-existing of events is more likely after a treatment (paired data)
I have some nominal (binary) data at two time points. The data structure are as follows:
participant
event A
event B
event C
event D
1 Before
TRUE
TRUE
TRUE
FALSE
2 Before
TRUE
FALSE
TRUE
FALSE
3 ...
0
votes
1
answer
26
views
Quantifying + comparing the effects of conditioning on Shannon entropy
I'm working on a project that compares how predictable/unpredictable individuals' actions are in terms of how they transition between actions. We consider their actions to part of a first-order Markov ...
2
votes
1
answer
144
views
Maximize Mutual Information between multiple ("overlapping") RVs
I'd like to maximize the sum of Mutual Information between a RV $X$ and $K$ out of $N$ possible RVs $Z_i$.
$$ \max \sum_{i \in K} \text{MI}(X, Z_i) $$
However, when I unfold the sum I get
$$ \sum_{i \...
2
votes
1
answer
262
views
What is the information plane theorem for an autoencoder neural network?
Slide 8 (about 19 minutes into the video) of the Stanford Seminar - Information Theory of Deep Learning, Naftali Tishby has the following (rather informally stated) theorem.
Theorem (Information ...
2
votes
2
answers
139
views
Surprisal in rankings
I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race.
Intuitively, in a race with 100 people, ...
2
votes
0
answers
35
views
How much do we learn about a random subset? [closed]
Suppose we sample the following two random variables, for some large integer $n$:
Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
0
votes
0
answers
70
views
Calculating mutual Information in absence of a token/string over a class
Equation:
MI(X,Ci) = ∑P(X,Ci) . log ( P(X,Ci) / P(X)⋅P(Ci) )
where, X is url, t is a url token and Ci is the i-th class will be either spam or ham
P(t,C) will be either the frequency that the word &...
1
vote
1
answer
73
views
Measure the Mutual Information between two variables that is also shared by a third variable
Is there a way to measure how much of the information that two variables share is also shared by a third variable?
Say that there are three variables $X$, $Y$, $Z$ and that I need to:
Predict $Y$ ...
2
votes
0
answers
58
views
Mutual Information between the mean of normally-distributed random variables and another variable
Given normally-distributed (possibly dependent) random variables $X_1 \dots X_n$, their mean $X=\frac{1}{n} \sum_iX_i$, and another discrete r.v. $Y$, can we relate $MI(X,Y)$ with the individual $MI(...
3
votes
1
answer
889
views
mutual information between discrete random variables and continuous variables
How does one compute mutual information (MI) between discrete random variables and continuous random variables ?
It's easy to compute MI for dataset with only discrete random variables. For dataset ...
1
vote
4
answers
121
views
Mutual Information larger than smaller one of both entropies?
Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
1
vote
0
answers
50
views
Why would I get a non-zero value for mutual information for one variable, say x_m (belonging to (x_1, ..., x_n)), to target y, if x_m is a constant?
I have n 'features', (x_1, ..., x_n) and a target variable y. I use sklearn's mutual information score for feature selection to determine which features are the most important for the target y. I get ...
0
votes
1
answer
57
views
Mutual Information for variables w/o any pairwise data
When playing with an [example for a mutual information matrix][1] I realized I actually do get results even for a pair of variables where there is not a single observation where both variables are ...