Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

K-nearest neighbors to estimate mutual information

I would like to use the mutual_info_regression object from scikit-learn to get a rough idea of how well any individual feature ...
David's user avatar
  • 1,206
0 votes
0 answers
27 views

Does maximizing log likelihood lead to maximizing mutual information?

In supervised learning, we usually maximize log-likelihood, for example, by minimizing cross-entropy loss. Let's say we have data $X$ and their respective labels $Y$. We train a model to output a ...
John Weligon's user avatar
0 votes
0 answers
30 views

Mutual information with convex combination of dependent variable and random noise

Suppose I have random variables $A, B, C$ where $B$ is statistically-independent of $A$ and $C$ (e.g. it is random noise). Consider scalars $0 < \alpha < \beta < 1$ and random variables $X = (...
Jimmy's user avatar
  • 1
0 votes
0 answers
18 views

Information Value Definition on Siddiqui "Credit Risk Scorecards" Book

I was studying about information value, and saw the definition on "Siddiqui, Credit Risk Scorecards". I'm searchin for some references on why the formula there is that way, and also for the ...
Roger Danilo Figlie's user avatar
1 vote
0 answers
27 views

Does Mutual Information by the power of 3 for the numerator really exist?

I am currently trying to measure collocational strength using Mutual Information (MI). MI gives an edge for exclusive and infrequent words. As stated by Brezina (2018), in measuring collocational ...
pindakazen's user avatar
0 votes
0 answers
6 views

What is the best statistical measurement for collocational analysis?

I am a beginner with statistics and corpus linguistics, so sorry in advance if my explanation have gaps in it. So I want to perform corpus linguistics collocational strength analysis on a given corpus,...
pindakazen's user avatar
1 vote
0 answers
10 views

Analytical expression of mutual information between a multivariate Gaussian and a binary class variable

The goal is to construct settings that allow analytical computations of the mutual information in the form: $$I(X; Y) = H(X) - H(X | Y)$$ I am wondering if that is ...
ajl123's user avatar
  • 287
1 vote
0 answers
89 views

Theoretically optimal "loss" function for continuous distributions? [closed]

Classification problems have the benefit of being discrete, so it's easy to calculate how much information your model has. For example, in LLMs your training data has inputs $\mathbf{x}_i$, a list of ...
programjames's user avatar
0 votes
0 answers
25 views

Query regarding MI and NMI in the context of image fusion

I am exploring Mutual Information (MI) and Normalized Mutual Information (NMI) in the context of image fusion. While reviewing various sources, it's often mentioned that Mutual Information's value ...
925678's user avatar
  • 1
0 votes
0 answers
17 views

Mutual information between a vector of values and a statistic computed on the vector

I understand that the mutual information between two vectors X, Y from two distributions gives a measure of uncertainty of X knowing information about Y. But is it possible to measure mutual ...
cryptomath's user avatar
2 votes
1 answer
45 views

Conditional mutual information $I(X;Y|Z)$

If there is a function $Y=f(Z)$, is the following mutual information equal to zero? \begin{align} I(X;Y|Z)=0 \end{align} Intuitively, it is correct. But how can we prove this?
Harry's user avatar
  • 227
0 votes
0 answers
23 views

a continuous analog for adjusted mutual information

I'm interested in comparing a set of electoral redistricting plans to, which can be conceived as partitions of an area. A measure like mutual information looks attractive. Let $U_i$ be a district from ...
fgregg's user avatar
  • 1,200
1 vote
0 answers
48 views

Markov chain data processing inequality

For a Markov chain $X \rightarrow Y \rightarrow Z$, we have the following data processing inequality: $I(Y;X) \ge I(Z;X)$. Now for the Markov chain, $(W,X) \rightarrow Y \rightarrow Z$, can we prove ...
Bhutum Banerjee's user avatar
2 votes
1 answer
48 views

What is the Mutual Information for one instance?

I'm computing mutual information for several features where one of my datasets has one instance. One instance is because of a specific filtering criterion I used. I'm using ...
Sachz's user avatar
  • 121
0 votes
1 answer
186 views

How to interpret a mutual information value less than 1

For a mutual information of two continuous variables (X and Y), I interpret a value of M.I. = 1 bit (i.e., 2^1 = 2 distinguishable levels), to mean the following: If I know any given value of X, I ...
smccain's user avatar
  • 659
0 votes
1 answer
38 views

Why is the WordPiece algorithm implemented according to the maximum mutual information?

WordPiece is a subword segmentation algorithm in the field of natural language processing. Different from BPE, WordPiece will select a pair with the largest mutual information to merge each time, and ...
korangar leo's user avatar
1 vote
0 answers
60 views

Using Copulas to find mutual information

I have two multidimensional datasets $X, Y$ of dimensions $m \times n$. Here $m$ is the successive measurements and $n$ is the data collected during each measurement. We can say each of $m$ are ...
amitha's user avatar
  • 117
0 votes
0 answers
48 views

Measuring the epistemic uncertainty mutual information

I possess training data denoted as $D$, a given test input denoted as $x$, and a model that is parameterized by a random variable $\theta$. This model is a neural network containing dropout layers, ...
rando's user avatar
  • 236
1 vote
1 answer
97 views

Negative Mutual information python

I calculated mutual information with the PyInform function with local = True MI = mutual_info(xs, ys, local=True) In blue you see MI between the yellow and green line. Can anyone tell me why MI ...
Ang's user avatar
  • 13
0 votes
1 answer
44 views

Jeffreys divergence for a normal bivariate

I am reading the book Information Theory and Statistics of S. Kullback. In page 8 ((4.3) it is shown that the KL divergence between the joint bivariate normal and the product of the corresponding ...
noosdev's user avatar
  • 11
1 vote
0 answers
44 views

Mutual Information between Gaussians in the limiting, strongly correlated case

In this question it is derived that if X, Y are correlated Gaussian variables, the mutual information between them is given by $$ I(X;Y) = \log\left(\frac{\det(\Sigma_X)\det(\Sigma_Y)}{\det\Sigma_{XY}}...
Maximilian Matthé's user avatar
0 votes
1 answer
72 views

How is the rate $H(X\mid Y)$ achieved in Slepian-Wolf coding?

Given two (generally correlated) sources $X,Y$, Slepian-Wolf coding is a protocol that shows it's possible to encode them separately, then have $X$ send $Y$ only $n H(X\mid Y)$ bits of information, ...
glS's user avatar
  • 373
0 votes
0 answers
23 views

Pointwise mutual information correction for negative bias

I am studying the point wise mutual information and I have understood that it has some limitations in NPL like the bias towards negative values (it tends to assign high scores to low-frequency events)....
Bibi's user avatar
  • 57
0 votes
0 answers
54 views

Can We Use Mutual Information to Determine if a Time Series has a Difference Trend or a Time Trend?

Let's say we have a finite real-valued time series (finite subsequence of a realization of a real-valued stochastic process), $X_t$. To address my question, we make no initial stationarity assumptions....
QMath's user avatar
  • 337
0 votes
0 answers
22 views

Is there a natural extension of the ACF/PACF to more general measures of dependency?

Assume that we have a time series that we want to model as a stationary real-valued stochastic process: $$X_t , t \in \{0, 1, \dots \}.$$ Two complementary measures of linear dependence between the ...
QMath's user avatar
  • 337
0 votes
1 answer
122 views

Finding trend similarity between two different time series

I am very new to this data analytics field, so bit confused about what to do. I have 2 time series datasets of two different products that have the same attributes. I want to find trend similarity ...
Divya Prakash's user avatar
1 vote
1 answer
108 views

Mutual information I(X, Y) >= I(f(X), f(Y)) for deterministic f?

I have the intuition that applying a deterministic function to a pair random variables cannot increase their mutual information, because the function can only decrease each of their entropies. I would ...
jcp's user avatar
  • 521
0 votes
0 answers
22 views

What do the authors mean by this, regarding estimation of mutual information

I found the following while reading a paper [1] and got confused: Replacing $k_{ij}$, $k{i.}$, respectively $k_{.j}$, by $k_{i,j}$, $k_{i.}$ and $k_{.j}$ provides us with estimates of entropy and ...
Chika's user avatar
  • 1
1 vote
0 answers
19 views

How do I numerically compute $I(X;CX+Y)$?

Given that $X\sim\text{Bernoulli}(\nu)$ for some $\nu\in(0,1)$, and $Y\sim N(0,1)$ are independent random variables. I want to compute the mutual information $I(X;CX+Y)$, where $C$ is some known non-...
Resu's user avatar
  • 229
0 votes
0 answers
36 views

Trying to understand the derivation in the Information Bottleneck Method

I'm trying to understand the proofs on the The information bottleneck method paper by Tishby, Pereira and Bialek without luck. In particular, the second term in the functional derivative of the ...
adn's user avatar
  • 103
0 votes
0 answers
25 views

Model marginal and joint distributions from a sample of unkown number of categories

To illustrate the problems imagine I'm drawing labelled spheres from a box. I may or may not know the number of spheres in the box (does it make a difference?) If I draw 10 spheres from the box and ...
jcp's user avatar
  • 521
1 vote
2 answers
317 views

Is mutual information defined for random variables that are vectors?

I know that mutual information is a measure for how similar two random variables are. There is plenty of information available about how to empirically determine mutual information empirically from ...
financial_physician's user avatar
0 votes
0 answers
24 views

What is the non-asymptotic counterpart of the mutual information?

The mutual information of a joint probability distribution $p(x,y)$ tells us, if we send $n$-letter messages with each letter drawn from the marginal distribution $p_X(x)$, that we can use roughly $2^{...
glS's user avatar
  • 373
1 vote
1 answer
530 views

Use Mutual Information Feature Selection For Categorical Feature

I have a dataset in which there are Features of both float and object type . I want to apply feature selection On this dataset in such a way that fisrt Find Mutual Information Score of all the ...
nasim_bb's user avatar
3 votes
1 answer
297 views

Mutual Information result in python for Feature selection unexpected result

I've started to study feature selection techniques and i have a situation that I don't understand. I've created a synthetic dataset with 5 predictor variables and a label, the predictor variables are ...
Guillermo Álvarez's user avatar
0 votes
0 answers
276 views

How to compute mutual information between two samples when some features are discrete and some are continuous?

If we need to compute mutual information between two features (columns) where each feature can only be either continuous or discrete, it's easy to compute mutual information between them using ...
user1769197's user avatar
  • 1,164
5 votes
1 answer
3k views

Scikit-learn: mutual info regression

My understanding of the mutual information between two random variables X and Y can be stated formally as follows: $$I(X ; Y) = H(X) — H(X | Y)$$ Where $I(X; Y)$ is the mutual information for $X$ and $...
user1769197's user avatar
  • 1,164
1 vote
0 answers
12 views

How to statistically test whether the co-existing of events is more likely after a treatment (paired data)

I have some nominal (binary) data at two time points. The data structure are as follows: participant event A event B event C event D 1 Before TRUE TRUE TRUE FALSE 2 Before TRUE FALSE TRUE FALSE 3 ...
William Wong's user avatar
0 votes
1 answer
26 views

Quantifying + comparing the effects of conditioning on Shannon entropy

I'm working on a project that compares how predictable/unpredictable individuals' actions are in terms of how they transition between actions. We consider their actions to part of a first-order Markov ...
ivak's user avatar
  • 1
2 votes
1 answer
144 views

Maximize Mutual Information between multiple ("overlapping") RVs

I'd like to maximize the sum of Mutual Information between a RV $X$ and $K$ out of $N$ possible RVs $Z_i$. $$ \max \sum_{i \in K} \text{MI}(X, Z_i) $$ However, when I unfold the sum I get $$ \sum_{i \...
N8_Coder's user avatar
  • 199
2 votes
1 answer
262 views

What is the information plane theorem for an autoencoder neural network?

Slide 8 (about 19 minutes into the video) of the Stanford Seminar - Information Theory of Deep Learning, Naftali Tishby has the following (rather informally stated) theorem. Theorem (Information ...
Galen's user avatar
  • 8,412
2 votes
2 answers
139 views

Surprisal in rankings

I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race. Intuitively, in a race with 100 people, ...
Alex I's user avatar
  • 1,123
2 votes
0 answers
35 views

How much do we learn about a random subset? [closed]

Suppose we sample the following two random variables, for some large integer $n$: Let random variable $X_1$ be a uniformly random subset of $m$ elements chosen from set $[n]:=\{1,\dots,n\}$, where $m ...
user367625's user avatar
0 votes
0 answers
70 views

Calculating mutual Information in absence of a token/string over a class

Equation: MI(X,Ci) = ∑P(X,Ci) . log ( P(X,Ci) / P(X)⋅P(Ci) ) where, X is url, t is a url token and Ci is the i-th class will be either spam or ham P(t,C) will be either the frequency that the word &...
Nouman Javaid's user avatar
1 vote
1 answer
73 views

Measure the Mutual Information between two variables that is also shared by a third variable

Is there a way to measure how much of the information that two variables share is also shared by a third variable? Say that there are three variables $X$, $Y$, $Z$ and that I need to: Predict $Y$ ...
Lay González's user avatar
2 votes
0 answers
58 views

Mutual Information between the mean of normally-distributed random variables and another variable

Given normally-distributed (possibly dependent) random variables $X_1 \dots X_n$, their mean $X=\frac{1}{n} \sum_iX_i$, and another discrete r.v. $Y$, can we relate $MI(X,Y)$ with the individual $MI(...
user1767774's user avatar
3 votes
1 answer
889 views

mutual information between discrete random variables and continuous variables

How does one compute mutual information (MI) between discrete random variables and continuous random variables ? It's easy to compute MI for dataset with only discrete random variables. For dataset ...
user1769197's user avatar
  • 1,164
1 vote
4 answers
121 views

Mutual Information larger than smaller one of both entropies?

Lot's of questions and good answers on mutual information e.g. here out there and I think I get the concept which is also nicely explained on wikipedia. But I'm nervous that R's infotheo package is ...
MarkH's user avatar
  • 197
1 vote
0 answers
50 views

Why would I get a non-zero value for mutual information for one variable, say x_m (belonging to (x_1, ..., x_n)), to target y, if x_m is a constant?

I have n 'features', (x_1, ..., x_n) and a target variable y. I use sklearn's mutual information score for feature selection to determine which features are the most important for the target y. I get ...
Socorro's user avatar
  • 71
0 votes
1 answer
57 views

Mutual Information for variables w/o any pairwise data

When playing with an [example for a mutual information matrix][1] I realized I actually do get results even for a pair of variables where there is not a single observation where both variables are ...
MarkH's user avatar
  • 197

1
2 3 4 5
8