Questions tagged [mutual-information]

mutual information is a concept from information theory. It is a measure of joint dependence between two random variables, which is not, like the usual correlation coefficient, limited to scalar variables.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Social Network Analysis: assortativity between two classes and network structure

I have a network where nodes are labelled in two classes "a" and "b". I want to measure how connected these two groups are and I looked at assortativity by group. I want to use this measure in more ...
0
votes
1answer
39 views

Mutual information between a vector and a constant

Let's say that we have a tensor x sampled from its probability distribution, and that we have a constant vector c sampled from a degenerate distribution (its value is constant). Is the mutual ...
0
votes
0answers
24 views

Mutual information estimated on subsets of data

I am estimating the mutual information for a continuous data set using the kNN-based mutual information estimator proposed by Kraskov et al [1]. Lets consider two features $X$ and $Y$, and the ...
2
votes
0answers
27 views

Interpretable General Measure of Dependence

I am looking for an interpretable measure between two random variables $X$ and $Y$ which quantifies the dependence between the two but does not assume linearity. Essentially, I am looking for a ...
0
votes
0answers
14 views

Advice on anomaly detection for a text

I have an idea of retreiving texts that have anomalous distribution of token frequency. Let's say I have a corpus of texts, and I build a document-term matrix based on token frequency. Naturally, ...
1
vote
0answers
16 views

The monotonicity of entropy operator

Define the entropy operator of a distribution as $\mathbb{H}(p) = -\int p \log p$, how does the entropy change for distributions that are proportional to the powers of $p$? For example, define $\...
3
votes
1answer
57 views

Mutual information between $X$ and $f(X)$

Let $X$ and $Y$ be two random variables, where $Y=f(X)$ is a deterministic function of $X$. Furthermore suppose $X,Y$ are continuous and that $f$ is smooth. Is the mutual information between $X$ and $...
2
votes
2answers
40 views

Rewriting the mutual information of a linear model through conditional expectation

I am reading the following paper: http://web.mit.edu/18.325/www/telatar_capacity.pdf In this paper we have the following linear model, with $\mathbf{n}$ being additive noise: \begin{equation} \...
2
votes
0answers
24 views

Mutual Information as a Uncertainty Reduction Criteria For Arbitrary Field

I have been reviewing some papers such as this that uses Mutual Information (MI) as a criteria to obtain most informative point to approximate some large field an reduce uncertainty of the field. This ...
2
votes
1answer
54 views

Difference between Covariance and Mutual Information

I was going through the posts that describe the difference between covariance and MI and came across following from Quora The covariance of two random variables measures the strength of the linear ...
1
vote
1answer
37 views

Understanding Multidimensional Mutual Information

Given random variables $\vec{x}, \vec{y} \in \mathbb{R}^n$, and the mutual information, defined as $I(\vec{x} : \vec{y}) = H(\vec{x}) + H(\vec{y}) - H(\vec{x}, \vec{y})$ is it true that $I(\vec{x}: ...
0
votes
0answers
21 views

Mutual Information between covariates and response in linear regression

Suppose we obtain a linear regression model through least square minimization: $$ \hat y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 $$ Suppose $x_1 \sim N(0, \sigma_1^2)$, $x_2 \sim Ber(0.5)$. Is it ...
2
votes
1answer
77 views

Categorical variables sklearn random forest

I'm a bit confused with the use of Random Forest in Sklearn in case we have categorical variables. I've read this article stating that one hot encoding affects performance negatively when using ...
2
votes
0answers
53 views

Law of total expectation applied to conditional mutual information?

I came across a book where the author uses the following property of mutual information: Let $X$,$Y$,$Z$ be arbitrary discrete random variables and let $W$ be an indicator random variable. $$ (1)\ \ ...
0
votes
0answers
11 views

Characterization of two gaussian mixtures with common part

I'm looking for either the covariance, the joint entropy, something to ultimately get the mutual information $I(X,Y)$ where $X$ and $Y$ are defined as : $X = c*\epsilon_c + \epsilon_X$ $Y = c*\...
2
votes
1answer
59 views

“Any meaning to the concept of ‘Self Mutual Information?”

** “Any meaning to the concept of ‘Self Mutual Information?” ** A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
0
votes
1answer
41 views

Selecting Feature weights

I use the knn Classifier for a binary classification problem. To improve the classification results I would like to multiply features by weights that are learned from data. I found different ways to ...
1
vote
0answers
31 views

Estimate joint probability of two dependent variable

I've a dataset created in the following way: The input of the system is a 8 bit binary number ranging from $x_1$= 0000 0000 to $x_N$ = 1111 1111 For every input i've read the output of the system (...
0
votes
0answers
76 views

how to derive $I(x,x)=H(X)$

Trying to understand how to derive that the self mutual information I(X,X) is equal to the entropy H(X), as stated on the wiki page wiki mutual information, "therefore H(X) = I(X;X)". More than that, ...
0
votes
0answers
13 views

Simple worked-out example to compute mutual information between two random variables that are vectors

Here is a simple worked out example to compute MI between two random variables $X,Y\in \mathbb{R}$. This also does a good job. Suppose now i am dealing with two random vectors $P\in \mathbb{R}^3,\ Q\...
1
vote
0answers
44 views

Can linear (Pearson) correlation overestimate the true value?

It's well known that Pearson correlation can underestimate the true value in case of non-linear relationship. But can it overestimate? For example, Pearson correlation of discontinuous distributions, ...
3
votes
0answers
48 views

How to interpret the relationship between MI score and correlation score [closed]

I am confusing about the relationship between MI score and correlation score. I've searched for some information and know that: ...
0
votes
1answer
31 views

Mutual information between subsets of variables in the multivariate normal distribution

Let $\vec{X}$ be a random vector following a multi-variate normal distribution $P(\vec X)$ with covariance matrix $\Sigma$ and zero means (for simplicity). Consider a partition of $\vec X$ into two ...
2
votes
0answers
51 views

A mathematical definition of noteworthiness

If you're a sport fan, you've probably come across these bogus, cherry-picked stats, things like: "This is the first time a player has scored 32 points on the Ides of March when the temperature was ...
0
votes
0answers
19 views

Mutual Information for monotonic function mapping

Consider a random Gaussian input $\mathbf{x}\in\mathbb{R}^n \sim \mathcal{N}(0,q\mathbf{I})$ as a input to a deterministic function mapping as $y= f(<\mathbf{w,x}>) $ for a monotonic function f(...
0
votes
1answer
148 views

Feature selection : Mutual information between 2 features or between feature and target?

I am aware that Mutual Information (MI) is one solution for feature reduction techniques. Consider a binary model with feature vector X = (x1, x2,...,xn) and ...
1
vote
0answers
20 views

Understanding Mutual Information score between two variables

I have two variables - a target and a predictor. I want to capture the relationship between the two, but they are non-normal distributed, so I can't use correlation. I remember a Data Science ...
0
votes
0answers
17 views

measure of the difference between variation of two time series of probability distributions

I'm looking at a series of particle density probabilities of proteins floating on a cell, these particles move around and also blink, so these density probabilities differ a bit from one time to ...
0
votes
0answers
16 views

Proof help for multivariate mutual information as a sum of entropies

I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies. $J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(...
0
votes
0answers
58 views

Who to attribute information gain to?

I am writing a paper where I examine information gain specifically with regards to feature selection and am wondering what the proper reference should be. I have looked all over and I can't find a ...
1
vote
0answers
39 views

Which has more mutual information with a multivariate Gaussian: its first principal component, or its first factor?

I have a $k$-dimensional Gaussian random variable $X\sim\mathcal{N}(0, \Sigma_X)$. What I want is a 1-dimensional scalar r.v. $Y\sim\mathcal{N}(0,1)$ that is jointly Gaussian with $X$ while maximizing ...
1
vote
0answers
71 views

How to show an alternate data processing inequality concerning KL divergence between conditionals?

Suppose $(Y,X) \sim F \in \mathcal{P(\mathbb{R^d})}$. Consider an arbitrary transformation $f$ that acts on $X$. My intuition is that the following should be a result in information theory: $$ \...
0
votes
0answers
28 views

Finding PCA-like directions in feature space that maximise sensitivity to a target variable

I have a fairly large space of feature variables in which I want to build a predictor for a target variable. My input dataset for training the predictor are sampled from the space using a mix of log ...
3
votes
2answers
66 views

Using conditional mutual information with measurements

I want to compute the mutual information between two discrete random variables $x_1,x_2$ whose PMFs are affected by measurements resp. $z_1,z_2$ (one measurement per variable) in a known way. Based on ...
0
votes
1answer
38 views

Need help understanding Toru Tsujishita' theorem on Triple Information

I struggling to understand point (ii) of Toru Tsujishita's theorem (here) on Triple- (Interaction- or Co-) Information What is meant with the maps $\varphi_j$ and $\varphi_k$? A biunivocal ...
3
votes
1answer
283 views

Confusion about proof in “Representation Learning with Contrastive Predictive Coding”

In the Appendix A.1 of the paper "Representation Learning with Contrastive Predictive Coding", the author prove $\log N-\mathcal L_N$ is the lower bound of mutual information between $x_{t+k}...
0
votes
0answers
29 views

Finding related variables

I have read this and this question, but non of them answers my question properly. I am not very familiar with concepts of stat, so please bear with me. I have 2 types of variables, type A (circles) ...
0
votes
0answers
64 views

How to choose sample size from probability density for computing mutual information based on continuous variables

I need to compute mutual information gain based two continuous variables $X$ and $Y$ $I(X|Y) = \int_X\int_Y p_{x.y}(x,y) \log(\frac{p_{x.y}(x,y)}{p_{x}(x)p_{y}(y)})$. I have used Kernel Density ...
1
vote
1answer
55 views

If $T$ is a sufficient statistic for $\theta$, is $H(\theta\mid x) = H(\theta\mid T(x))$?

I was trying to prove that sufficient statistics attain equality in the data processing inequality by a slightly different route than I usually see, and came across an odd expression. (I care more ...
1
vote
0answers
129 views

Non-negativity of interaction information for special trivariate case

Consider a discrete trivariate distribution $P(X_1, X_2, Y)$, which satisfies $$ p(x_1, x_2, y) = \min( p(x_1,y), p(x_2,y) ), $$ for all $x_1$ and $x_2$ for which $p(x_1, x_2) > 0$ and for all ...
0
votes
1answer
41 views

Mutual Information between multi-dimensional and single dimensional Variables

I would like to estimate MI between two variables X and Y of shape (nXd) and ...
0
votes
1answer
183 views

Information Bottleneck principle of Deep Learning model implementation

I am trying to implement Information Bottleneck principle. In which one of the observation is Mutual Information between Input data X and Hidden layer's out H keeps reducing as we go deeper. In other ...
0
votes
1answer
39 views

If $A=B∥C$, and $PMI(B;C)=0$, and $P(B)=P(C)$, then how is it possible that $PMI(A;B)=PMI(A;C)=PMI(A)$?

Consider this simple boolean relationship between the binary variables A, B and C: $$A=B∥C$$ I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0. We also have these extra ...
0
votes
0answers
63 views

Choosing appropriate statistical tests

I am trying to run stats on medical treatment data and would like your guys and girls help Goal: Comparing two trends of annual percentages for statistical significance. Is one decreasing due to the ...
0
votes
1answer
149 views

How to normalize mutual information between to real-valued random variables?

How can I normalize mutual information between to real-valued random variables using Python or R? ...
0
votes
1answer
113 views
0
votes
1answer
171 views

How to calculate mutual information between a feature and target variable?

Mutual information measures how much information the distribution of one variable provides about the distribution of another variable. In my case, I have samples of a feature variable $X \in \mathbb{...
1
vote
2answers
299 views

Score of importance from feature selection techniques

Can I get the score of importance for each feature in feature selection methos such as Chi2, Information Gain (IG), or Recursive Feature Elimination (RFE)? Or they just provide a list of important ...
1
vote
0answers
26 views

Is there an archive of closed-form mutual information among the “famous” distributions?

I'm looking for a document or compilation table of closed-form mutual information as a function of their parameters, for known distributions such as normal, gamma, Poisson distributions. At least, I ...
6
votes
1answer
147 views

Why not normalizing mutual information with harmonic mean of entropies?

This is a similar question to this one (which has unfortunately no answer yet), although I believe my question is more specific. Let $X$ and $Y$ be two discrete random variables with outcome space, ...

1
2 3 4 5 6